[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:26 dist-test-slave-zkpd krb5kdc[31368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:26 dist-test-slave-zkpd krb5kdc[31368](info): set up 2 sockets
Aug 08 02:22:26 dist-test-slave-zkpd krb5kdc[31368](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 02:22:27.652438 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:27 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:27.770545 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40095
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:40095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:27.852890 31381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:27.853119 31381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:27.853157 31381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:27.854303 31381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:27.854370 31381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:27.854396 31381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:27.854427 31381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:27.854451 31381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:27.855998 31381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:40095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:27.856259 31381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:27.856492 31381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:27.861722 31386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:27.861783 31389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:27.861838 31381 server_base.cc:1034] running on GCE node
W20260808 02:22:27.861768 31387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:27.862118 31381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:27.862372 31381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:27.863547 31381 hybrid_clock.cc:648] HybridClock initialized: now 1786155747863533 us; error 30 us; skew 500 ppm
Aug 08 02:22:27 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155747, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:27.866083 31381 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:27.866499 31381 webserver.cc:460] Webserver started at http://127.30.160.62:38567/ using document root <none> and password file <none>
I20260808 02:22:27.866714 31381 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:27.866778 31381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:27.866926 31381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:27.868042 31381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2"
format_stamp: "Formatted at 2026-08-08 02:22:27 on dist-test-slave-zkpd"
server_key: "8fbb68576e169937568bdb1ee73a73c5"
server_key_iv: "26b52e56cc970096ff201b26bd11c062"
server_key_version: "encryptionkey@0"
I20260808 02:22:27.868427 31381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2"
format_stamp: "Formatted at 2026-08-08 02:22:27 on dist-test-slave-zkpd"
server_key: "8fbb68576e169937568bdb1ee73a73c5"
server_key_iv: "26b52e56cc970096ff201b26bd11c062"
server_key_version: "encryptionkey@0"
I20260808 02:22:27.869772 31381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:27.870535 31397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:27.870723 31381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:27.870822 31381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2"
format_stamp: "Formatted at 2026-08-08 02:22:27 on dist-test-slave-zkpd"
server_key: "8fbb68576e169937568bdb1ee73a73c5"
server_key_iv: "26b52e56cc970096ff201b26bd11c062"
server_key_version: "encryptionkey@0"
I20260808 02:22:27.870901 31381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:27.894917 31381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:27.896943 31381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:27.897109 31381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:27.901096 31381 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:40095
I20260808 02:22:27.901130 31451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:40095 every 8 connection(s)
I20260808 02:22:27.901479 31381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:27.902376 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:27.904714 31452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2: Bootstrap starting.
I20260808 02:22:27.905303 31452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:27.905503 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 31381
I20260808 02:22:27.905614 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:27.905637 31452 log.cc:824] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:27.906229 31360 external_mini_cluster.cc:1428] Setting key a591427d443cb31d7ca1f134cd1059ef
I20260808 02:22:27.906420 31452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2: No bootstrap required, opened a new log
I20260808 02:22:27.908425 31452 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40095 } }
I20260808 02:22:27.908560 31452 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:27.908583 31452 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cb801d4ef640a2ab0a5a8c31b456e2, State: Initialized, Role: FOLLOWER
I20260808 02:22:27.908713 31452 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40095 } }
I20260808 02:22:27.908810 31452 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:27.908855 31452 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:27.908903 31452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:27.909723 31452 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40095 } }
I20260808 02:22:27.909854 31452 leader_election.cc:302] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8cb801d4ef640a2ab0a5a8c31b456e2; no voters:
I20260808 02:22:27.910038 31452 leader_election.cc:288] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:27.910061 31458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:27.910168 31458 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [term 1 LEADER]: Becoming Leader. State: Replica: f8cb801d4ef640a2ab0a5a8c31b456e2, State: Running, Role: LEADER
I20260808 02:22:27.910248 31458 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40095 } }
I20260808 02:22:27.910295 31452 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:27.910569 31458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40095 } } }
I20260808 02:22:27.910638 31458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:27.910770 31458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8cb801d4ef640a2ab0a5a8c31b456e2. Latest consensus state: current_term: 1 leader_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb801d4ef640a2ab0a5a8c31b456e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40095 } } }
I20260808 02:22:27.910851 31458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2 [sys.catalog]: This master's current role is: LEADER
W20260808 02:22:27.912153 31476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:27.912210 31476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:27.912256 31463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:27.912418 31463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:27.914052 31463 catalog_manager.cc:1347] Generated new cluster ID: dc916f11588949c4be6c6496629bf88f
I20260808 02:22:27.914111 31463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:27 dist-test-slave-zkpd krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:27.924230 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:27.907155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.30.160.62:40095)
0808 02:22:27.907336 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:27.907345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:27.907591 (+ 246us) server_negotiation.cc:407] Connection header received
0808 02:22:27.909588 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:27.909592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:27.909632 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:27.909764 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:27.911968 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:27.913209 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:27.916483 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:27.916892 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:27.917013 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:27.920396 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:27.920400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:27.920400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:27.920410 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:27.922212 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:27.922657 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:27.922665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:27.922666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:27.922748 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:27.923023 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:27.923024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:27.923024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:27.923264 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:27.923451 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:27.923719 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 02:22:27.923867 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260808 02:22:27.938169 31463 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:27.938786 31463 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:27.943603 31463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8cb801d4ef640a2ab0a5a8c31b456e2: Generated new TSK 0
I20260808 02:22:27.943763 31463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:27.980530 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40095
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:28.095541 31481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:28.095739 31481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:28.095763 31481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:28.096932 31481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:28.096975 31481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:28.097035 31481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:28.098462 31481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40095
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:28.098713 31481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:28.098948 31481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:28.103778 31487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:28.103801 31486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:28.103786 31489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:28.104144 31481 server_base.cc:1034] running on GCE node
I20260808 02:22:28.104391 31481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:28.104666 31481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:28.105834 31481 hybrid_clock.cc:648] HybridClock initialized: now 1786155748105818 us; error 43 us; skew 500 ppm
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:28.108355 31481 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:28.108768 31481 webserver.cc:460] Webserver started at http://127.30.160.1:34617/ using document root <none> and password file <none>
I20260808 02:22:28.108999 31481 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:28.109066 31481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:28.109197 31481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:28.110271 31481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "02bd8c0bae59433fa8381f34c0e2699f"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "88def779e3b085461055b195cc9cc857"
server_key_iv: "fbc2ca13bdb2358c96fcb6f1060bb41e"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.110633 31481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "02bd8c0bae59433fa8381f34c0e2699f"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "88def779e3b085461055b195cc9cc857"
server_key_iv: "fbc2ca13bdb2358c96fcb6f1060bb41e"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.112272 31481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:28.113019 31496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.113193 31481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:28.113296 31481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "02bd8c0bae59433fa8381f34c0e2699f"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "88def779e3b085461055b195cc9cc857"
server_key_iv: "fbc2ca13bdb2358c96fcb6f1060bb41e"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.113384 31481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:28.128021 31481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:28.129812 31481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:28.129974 31481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:28.130204 31481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:28.130506 31481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:28.130561 31481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.130604 31481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:28.130631 31481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.136708 31481 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:38679
I20260808 02:22:28.136756 31614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:38679 every 8 connection(s)
I20260808 02:22:28.137056 31481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:28.146011 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 31481
I20260808 02:22:28.146116 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:28.146274 31360 external_mini_cluster.cc:1428] Setting key a2f4dd53c99aaf6c3a7f9bbfe6b6e27d
I20260808 02:22:28.146337 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.137733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:50773 (local address 127.30.160.62:40095)
0808 02:22:28.137803 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:28.137806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:28.138033 (+ 227us) server_negotiation.cc:407] Connection header received
0808 02:22:28.138561 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:28.138563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:28.138580 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:28.138643 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:28.139593 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.140197 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.141189 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.141410 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.141486 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.143541 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:28.143543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:28.143544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:28.143551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:28.145052 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.145419 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.145420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.145420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.145473 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.145684 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.145684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.145685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.145782 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:28.145891 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:28.146139 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 02:22:28.146243 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:22:28.146543 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.137716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40095 (local address 127.30.160.1:50773)
0808 02:22:28.137912 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:28.137948 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:28.138455 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:28.138719 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:28.138723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:28.139297 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:28.139491 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:28.139496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.140309 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.140310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:28.141091 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:28.141093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.141507 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.141509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:28.141628 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.142044 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:28.142052 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:28.143435 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:28.145125 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:28.145133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:28.145134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:28.145351 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:28.145532 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:28.145533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:28.145533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:28.145628 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:28.145889 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:28.145893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:28.146000 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 02:22:28.146111 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:22:28.146264 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260808 02:22:28.147060 31615 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40095
I20260808 02:22:28.147157 31615 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:28.147349 31615 heartbeater.cc:511] Master 127.30.160.62:40095 requested a full tablet report, sending...
I20260808 02:22:28.147955 31413 ts_manager.cc:194] Registered new tserver with Master: 02bd8c0bae59433fa8381f34c0e2699f (127.30.160.1:38679)
I20260808 02:22:28.148581 31413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:50773
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:28.171876 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40095
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:28.255039 31621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:28.255226 31621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:28.255249 31621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:28.256397 31621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:28.256435 31621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:28.256461 31621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:28.257906 31621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40095
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:28.258145 31621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:28.258389 31621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:28.263299 31626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:28.263374 31627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:28.263337 31629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:28.263535 31621 server_base.cc:1034] running on GCE node
I20260808 02:22:28.263738 31621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:28.264020 31621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:28.265164 31621 hybrid_clock.cc:648] HybridClock initialized: now 1786155748265150 us; error 29 us; skew 500 ppm
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:28.268004 31621 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:28.268440 31621 webserver.cc:460] Webserver started at http://127.30.160.2:32937/ using document root <none> and password file <none>
I20260808 02:22:28.268672 31621 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:28.268739 31621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:28.268895 31621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:28.269937 31621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "7943722821e94370928518507996aa46"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "3015e51bd1aa83c53c0e8ef2e2edccaf"
server_key_iv: "0f4176695ec6c35c9c7e82131dc33f41"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.270292 31621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "7943722821e94370928518507996aa46"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "3015e51bd1aa83c53c0e8ef2e2edccaf"
server_key_iv: "0f4176695ec6c35c9c7e82131dc33f41"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.271620 31621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:28.272431 31636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.272600 31621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.272701 31621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "7943722821e94370928518507996aa46"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "3015e51bd1aa83c53c0e8ef2e2edccaf"
server_key_iv: "0f4176695ec6c35c9c7e82131dc33f41"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.272807 31621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:28.297771 31621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:28.299804 31621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:28.300021 31621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:28.300300 31621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:28.300685 31621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:28.300722 31621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.300745 31621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:28.300758 31621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.306826 31621 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:37751
I20260808 02:22:28.306864 31752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:37751 every 8 connection(s)
I20260808 02:22:28.307202 31621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:28.316236 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.308036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:37563 (local address 127.30.160.62:40095)
0808 02:22:28.308106 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:28.308108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:28.308290 (+ 182us) server_negotiation.cc:407] Connection header received
0808 02:22:28.308799 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:28.308801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:28.308813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:28.308878 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:28.309774 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.310292 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.311411 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.311627 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.311697 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.313819 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:28.313820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:28.313821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:28.313827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:28.315011 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.315379 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.315380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.315381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.315432 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.315645 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.315646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.315646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.315743 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:28.315794 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:28.316098 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 02:22:28.316168 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:22:28.316473 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.307993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40095 (local address 127.30.160.2:37563)
0808 02:22:28.308157 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:28.308200 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:28.308692 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:28.308936 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:28.308939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:28.309502 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:28.309697 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:28.309702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.310363 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.310364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:28.311248 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:28.311251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.311714 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.311715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:28.311827 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.312327 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:28.312336 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:28.313723 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:28.315085 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:28.315091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:28.315093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:28.315315 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:28.315494 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:28.315494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:28.315495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:28.315591 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:28.315798 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:28.315803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:28.315963 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 02:22:28.316068 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:22:28.316186 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260808 02:22:28.317063 31753 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40095
I20260808 02:22:28.317090 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 31621
I20260808 02:22:28.317180 31753 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:28.317200 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:28.317400 31360 external_mini_cluster.cc:1428] Setting key 1a3fcf31fb80a9ef1624a4d8c8c7e685
I20260808 02:22:28.317404 31753 heartbeater.cc:511] Master 127.30.160.62:40095 requested a full tablet report, sending...
I20260808 02:22:28.317828 31414 ts_manager.cc:194] Registered new tserver with Master: 7943722821e94370928518507996aa46 (127.30.160.2:37751)
I20260808 02:22:28.318217 31414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:37563
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:28.342207 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40095
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:28.425397 31758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:28.425582 31758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:28.425604 31758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:28.426801 31758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:28.426846 31758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:28.426896 31758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:28.428421 31758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40095
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:28.428650 31758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:28.428874 31758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:28.433717 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:28.433801 31758 server_base.cc:1034] running on GCE node
W20260808 02:22:28.433727 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:28.433754 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:28.434147 31758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:28.434387 31758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:28.435552 31758 hybrid_clock.cc:648] HybridClock initialized: now 1786155748435539 us; error 29 us; skew 500 ppm
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:28.438097 31758 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:28.438506 31758 webserver.cc:460] Webserver started at http://127.30.160.3:33671/ using document root <none> and password file <none>
I20260808 02:22:28.438740 31758 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:28.438817 31758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:28.438966 31758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:28.439996 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "42fb2fabde1d4c199797aec79f6b1f3c"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "1e10417ab6e2689bb1f45feec07dee1a"
server_key_iv: "81c0e644b568bf2e1a8a7294353ece9b"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.440349 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "42fb2fabde1d4c199797aec79f6b1f3c"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "1e10417ab6e2689bb1f45feec07dee1a"
server_key_iv: "81c0e644b568bf2e1a8a7294353ece9b"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.441634 31758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:28.442350 31774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.442517 31758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:28.442615 31758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "42fb2fabde1d4c199797aec79f6b1f3c"
format_stamp: "Formatted at 2026-08-08 02:22:28 on dist-test-slave-zkpd"
server_key: "1e10417ab6e2689bb1f45feec07dee1a"
server_key_iv: "81c0e644b568bf2e1a8a7294353ece9b"
server_key_version: "encryptionkey@0"
I20260808 02:22:28.442689 31758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:28.449234 31758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:28.450958 31758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:28.451112 31758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:28.451326 31758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:28.451617 31758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:28.451670 31758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.451711 31758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:28.451740 31758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:28.457794 31758 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:34999
I20260808 02:22:28.457839 31889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:34999 every 8 connection(s)
I20260808 02:22:28.458171 31758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:28.466854 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.458838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:57329 (local address 127.30.160.62:40095)
0808 02:22:28.458881 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:28.458883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:28.459194 (+ 311us) server_negotiation.cc:407] Connection header received
0808 02:22:28.459691 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:28.459692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:28.459706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:28.459760 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:28.460741 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.461258 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.462257 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.462452 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.462525 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.464515 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:28.464516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:28.464517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:28.464523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:28.465673 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.466033 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.466034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.466034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.466084 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.466299 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.466300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.466300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.466398 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:28.466448 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:28.466674 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 02:22:28.466735 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:22:28.466952 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.458839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40095 (local address 127.30.160.3:57329)
0808 02:22:28.459044 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:28.459077 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:28.459584 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:28.459823 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:28.459827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:28.460460 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:28.460639 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:28.460643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.461361 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.461362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:28.462156 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:28.462158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.462533 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.462534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:28.462656 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.463050 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:28.463057 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:28.464432 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:28.465743 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:28.465750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:28.465752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:28.465968 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:28.466145 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:28.466146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:28.466146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:28.466244 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:28.466458 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:28.466462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:28.466568 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 02:22:28.466645 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:22:28.466759 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260808 02:22:28.466952 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 31758
I20260808 02:22:28.467031 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:28.467199 31360 external_mini_cluster.cc:1428] Setting key 343a6b509cc842b19bde75c4ea57c430
I20260808 02:22:28.467470 31890 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40095
I20260808 02:22:28.467577 31890 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:28.467759 31890 heartbeater.cc:511] Master 127.30.160.62:40095 requested a full tablet report, sending...
I20260808 02:22:28.468216 31414 ts_manager.cc:194] Registered new tserver with Master: 42fb2fabde1d4c199797aec79f6b1f3c (127.30.160.3:34999)
I20260808 02:22:28.468590 31414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:57329
I20260808 02:22:28.479187 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:28.502748 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.495142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.30.160.1:38679)
0808 02:22:28.495374 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:28.495379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:28.495390 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:28.495418 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:28.495420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:28.495433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:28.495501 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:28.496088 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.496810 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.497505 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.497811 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.497864 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.500019 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:28.500035 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:28.500036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:28.500042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:28.501488 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.501857 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.501864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.501865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.501917 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.502140 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.502141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.502141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.502303 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:28.502480 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:28.502531 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 02:22:28.502625 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
W20260808 02:22:28.503121 31526 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:28.524974 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:28.518486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.30.160.1:38679)
0808 02:22:28.518564 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:28.518566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:28.518611 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:22:28.518653 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:28.518654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:28.518664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:28.518711 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:28.519254 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.519721 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.520462 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:28.520650 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:28.520701 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:28.522746 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:28.522747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:28.522748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:28.522754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:28.524088 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.524394 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.524395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.524395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.524443 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:28.524603 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:28.524604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:28.524604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:28.524688 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:28.524775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:28.524850 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:22:28.524910 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:22:28.525501 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 31481
I20260808 02:22:28.531756 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 31621
I20260808 02:22:28.537343 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 31758
I20260808 02:22:28.542362 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 31381
2026-08-08T02:22:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1956 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31905](info): set up 2 sockets
Aug 08 02:22:28 dist-test-slave-zkpd krb5kdc[31905](info): commencing operation
krb5kdc: starting...
W20260808 02:22:29.599468 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:29 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:29.718274 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40901
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:29.801440 31919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:29.801663 31919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:29.801702 31919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:29.802826 31919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:29.802892 31919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:29.802919 31919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:29.802950 31919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:29.802976 31919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:29.804502 31919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:40901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:29.804765 31919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:29.804988 31919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:29.809729 31924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:29.809804 31919 server_base.cc:1034] running on GCE node
W20260808 02:22:29.809782 31925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:29.809780 31927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:29.810197 31919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:29.810511 31919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:29.811677 31919 hybrid_clock.cc:648] HybridClock initialized: now 1786155749811662 us; error 26 us; skew 500 ppm
Aug 08 02:22:29 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155749, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:29.818234 31919 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:29.818696 31919 webserver.cc:460] Webserver started at http://127.30.160.62:35013/ using document root <none> and password file <none>
I20260808 02:22:29.818905 31919 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:29.818946 31919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:29.819036 31919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:29.820111 31919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "ed9558ce65fb43f389d77d132c9119ec"
format_stamp: "Formatted at 2026-08-08 02:22:29 on dist-test-slave-zkpd"
server_key: "a97d52d47c169bc9673af436846b6aa9"
server_key_iv: "ad4ce8a79df37b23e1f93037707bb556"
server_key_version: "encryptionkey@0"
I20260808 02:22:29.820437 31919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "ed9558ce65fb43f389d77d132c9119ec"
format_stamp: "Formatted at 2026-08-08 02:22:29 on dist-test-slave-zkpd"
server_key: "a97d52d47c169bc9673af436846b6aa9"
server_key_iv: "ad4ce8a79df37b23e1f93037707bb556"
server_key_version: "encryptionkey@0"
I20260808 02:22:29.821704 31919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:29.822386 31936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:29.822575 31919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:29.822703 31919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "ed9558ce65fb43f389d77d132c9119ec"
format_stamp: "Formatted at 2026-08-08 02:22:29 on dist-test-slave-zkpd"
server_key: "a97d52d47c169bc9673af436846b6aa9"
server_key_iv: "ad4ce8a79df37b23e1f93037707bb556"
server_key_version: "encryptionkey@0"
I20260808 02:22:29.822773 31919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:29.836192 31919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:29.838399 31919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:29.838508 31919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:29.842310 31919 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:40901
I20260808 02:22:29.842379 31989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:40901 every 8 connection(s)
I20260808 02:22:29.842702 31919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:29.843035 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 31919
I20260808 02:22:29.843130 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:29.843294 31360 external_mini_cluster.cc:1428] Setting key 835778fe563cb1e34d10de1cae414083
I20260808 02:22:29.843747 31990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:29.846442 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec: Bootstrap starting.
I20260808 02:22:29.847193 31990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:29.847501 31990 log.cc:824] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:29.848296 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec: No bootstrap required, opened a new log
Aug 08 02:22:29 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:29.850592 31990 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9558ce65fb43f389d77d132c9119ec" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40901 } }
I20260808 02:22:29.850749 31990 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:29.850822 31990 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9558ce65fb43f389d77d132c9119ec, State: Initialized, Role: FOLLOWER
I20260808 02:22:29.850984 31990 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9558ce65fb43f389d77d132c9119ec" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40901 } }
I20260808 02:22:29.851106 31990 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:29.851145 31990 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:29.851202 31990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:29.851985 31990 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9558ce65fb43f389d77d132c9119ec" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40901 } }
I20260808 02:22:29.852094 31990 leader_election.cc:302] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed9558ce65fb43f389d77d132c9119ec; no voters:
I20260808 02:22:29.852255 31990 leader_election.cc:288] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:29.852314 31995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:29.852510 31995 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [term 1 LEADER]: Becoming Leader. State: Replica: ed9558ce65fb43f389d77d132c9119ec, State: Running, Role: LEADER
I20260808 02:22:29.852520 31990 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:29.852702 31995 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9558ce65fb43f389d77d132c9119ec" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40901 } }
I20260808 02:22:29.853938 31996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed9558ce65fb43f389d77d132c9119ec. Latest consensus state: current_term: 1 leader_uuid: "ed9558ce65fb43f389d77d132c9119ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9558ce65fb43f389d77d132c9119ec" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40901 } } }
I20260808 02:22:29.853945 31997 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed9558ce65fb43f389d77d132c9119ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9558ce65fb43f389d77d132c9119ec" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40901 } } }
I20260808 02:22:29.854156 31996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:29.854167 31997 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec [sys.catalog]: This master's current role is: LEADER
W20260808 02:22:29.854681 32010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:29.854825 32010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:29.854871 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:29.843830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.30.160.62:40901)
0808 02:22:29.844038 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:29.844048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:29.844083 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:22:29.844502 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:29.844508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:29.844551 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:29.844696 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:29.845295 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:29.846164 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:29.847328 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:29.847862 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:29.848025 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:29.850535 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:29.850539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:29.850540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:29.850551 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:29.852630 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:29.853052 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:29.853055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:29.853055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:29.853164 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:29.853418 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:29.853419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:29.853419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:29.853723 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:29.854380 (+ 657us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:29.854455 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:22:29.854593 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
I20260808 02:22:29.854892 32011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:29.855055 32011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:29.856609 32011 catalog_manager.cc:1347] Generated new cluster ID: 573a6279e20e47288e04bf13a7b38fc9
I20260808 02:22:29.856668 32011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:29.894438 32011 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:29.895169 32011 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:29.899426 32011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed9558ce65fb43f389d77d132c9119ec: Generated new TSK 0
I20260808 02:22:29.899622 32011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:29.934412 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40901
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:30.016844 32016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:30.017082 32016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:30.017125 32016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:30.018290 32016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:30.018361 32016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:30.018410 32016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:30.020004 32016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:30.020254 32016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:30.020488 32016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:30.025681 32021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:30.025681 32022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:30.025945 32016 server_base.cc:1034] running on GCE node
W20260808 02:22:30.025712 32024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:30.026284 32016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:30.026587 32016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:30.027750 32016 hybrid_clock.cc:648] HybridClock initialized: now 1786155750027732 us; error 47 us; skew 500 ppm
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:30.030293 32016 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:30.030730 32016 webserver.cc:460] Webserver started at http://127.30.160.1:35533/ using document root <none> and password file <none>
I20260808 02:22:30.030956 32016 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:30.031034 32016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:30.031185 32016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:30.032308 32016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "d534610f7257a3355bc6ddec528a4a51"
server_key_iv: "19552859540c17ae1dc23571b6ecd848"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.032687 32016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "d534610f7257a3355bc6ddec528a4a51"
server_key_iv: "19552859540c17ae1dc23571b6ecd848"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.034050 32016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:30.034832 32032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.035135 32016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:30.035240 32016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "d534610f7257a3355bc6ddec528a4a51"
server_key_iv: "19552859540c17ae1dc23571b6ecd848"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.035315 32016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:30.055526 32016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:30.057502 32016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:30.057668 32016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:30.057927 32016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:30.058245 32016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:30.058300 32016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.058341 32016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:30.058367 32016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.064425 32016 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:40307
I20260808 02:22:30.064479 32151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:40307 every 8 connection(s)
I20260808 02:22:30.064797 32016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:30.070003 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 32016
I20260808 02:22:30.070117 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:30.070333 31360 external_mini_cluster.cc:1428] Setting key ff1e4b25587d891f71ecf7c678a0607b
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:30.074631 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.065540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:34961 (local address 127.30.160.62:40901)
0808 02:22:30.065589 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.065591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.065799 (+ 208us) server_negotiation.cc:407] Connection header received
0808 02:22:30.066744 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.066744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.066754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.066814 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.067394 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.067927 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.068884 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.069079 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.069132 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.071378 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.071379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.071380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.071387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.073200 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.073617 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.073619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.073620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.073684 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.073968 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.073969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.073969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.074088 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.074165 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.074441 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.074538 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:22:30.074867 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.065492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40901 (local address 127.30.160.1:34961)
0808 02:22:30.065664 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.065708 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.066653 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.066869 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.066872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.067127 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:30.067319 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.067324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.068006 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.068006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.068804 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.068806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.069158 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.069159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.069256 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.069695 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:30.069704 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:30.071267 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:30.073286 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:30.073293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:30.073296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:30.073524 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:30.073763 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:30.073764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:30.073764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:30.073889 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:30.074156 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:30.074164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:30.074305 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.074419 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.074575 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260808 02:22:30.075474 32152 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40901
I20260808 02:22:30.075591 32152 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:30.075790 32152 heartbeater.cc:511] Master 127.30.160.62:40901 requested a full tablet report, sending...
I20260808 02:22:30.076429 31954 ts_manager.cc:194] Registered new tserver with Master: 0fe5f5870bb2411fb2fc5d034ad6b3ce (127.30.160.1:40307)
I20260808 02:22:30.077430 31954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:34961
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:30.094988 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40901
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:30.178112 32158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:30.178325 32158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:30.178351 32158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:30.179534 32158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:30.179576 32158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:30.179603 32158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:30.181097 32158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:30.181288 32158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:30.181468 32158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:30.186499 32166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:30.186518 32164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:30.186568 32163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:30.186551 32158 server_base.cc:1034] running on GCE node
I20260808 02:22:30.186913 32158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:30.187121 32158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:30.188369 32158 hybrid_clock.cc:648] HybridClock initialized: now 1786155750188338 us; error 41 us; skew 500 ppm
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:30.191083 32158 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:30.191526 32158 webserver.cc:460] Webserver started at http://127.30.160.2:44713/ using document root <none> and password file <none>
I20260808 02:22:30.191771 32158 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:30.191840 32158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:30.192018 32158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:30.193132 32158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "cb4795f882fa4c99a39c048bf5a2e479"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "256c77dae42b2476f811f2ce5b43a56b"
server_key_iv: "57e24ca0380ba368b23c04bbe3444852"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.193509 32158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "cb4795f882fa4c99a39c048bf5a2e479"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "256c77dae42b2476f811f2ce5b43a56b"
server_key_iv: "57e24ca0380ba368b23c04bbe3444852"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.194849 32158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:30.195565 32173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.195762 32158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:30.195866 32158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "cb4795f882fa4c99a39c048bf5a2e479"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "256c77dae42b2476f811f2ce5b43a56b"
server_key_iv: "57e24ca0380ba368b23c04bbe3444852"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.195982 32158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:30.213749 32158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:30.216156 32158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:30.216312 32158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:30.216547 32158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:30.216881 32158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:30.216934 32158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.216979 32158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:30.217000 32158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.222997 32158 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:33457
I20260808 02:22:30.223100 32291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:33457 every 8 connection(s)
I20260808 02:22:30.223403 32158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:30.230180 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 32158
I20260808 02:22:30.230307 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:30.230492 31360 external_mini_cluster.cc:1428] Setting key 0f465df0ce010e5cd23bd8e471698f41
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:30.234382 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.224289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:57127 (local address 127.30.160.62:40901)
0808 02:22:30.224429 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.224431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.224480 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:22:30.225578 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.225579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.225591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.225643 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.226349 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.227223 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.228463 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.228683 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.228765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.231104 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.231106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.231106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.231113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.232833 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.233300 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.233302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.233302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.233372 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.233643 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.233645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.233645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.233789 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.233863 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.234183 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.234281 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260808 02:22:30.234608 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.224176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40901 (local address 127.30.160.2:57127)
0808 02:22:30.224371 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.224410 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.225469 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.225711 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.225715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.226001 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:30.226270 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.226276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.227365 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.227367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.228377 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.228380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.228766 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.228767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.228885 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.229405 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:30.229414 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:30.231004 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:30.232943 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:30.232952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:30.232955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:30.233206 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:30.233445 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:30.233446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:30.233446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:30.233571 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:30.233868 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:30.233875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:30.234035 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.234149 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.234342 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260808 02:22:30.235210 32292 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40901
I20260808 02:22:30.235345 32292 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:30.235543 32292 heartbeater.cc:511] Master 127.30.160.62:40901 requested a full tablet report, sending...
I20260808 02:22:30.236006 31954 ts_manager.cc:194] Registered new tserver with Master: cb4795f882fa4c99a39c048bf5a2e479 (127.30.160.2:33457)
I20260808 02:22:30.236403 31954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:57127
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:30.258085 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40901
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:30.341765 32298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:30.342000 32298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:30.342037 32298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:30.343191 32298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:30.343257 32298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:30.343303 32298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:30.344905 32298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:30.345176 32298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:30.345418 32298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:30.350569 32304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:30.350564 32308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:30.350580 32303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:30.350921 32298 server_base.cc:1034] running on GCE node
I20260808 02:22:30.351154 32298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:30.351402 32298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:30.352546 32298 hybrid_clock.cc:648] HybridClock initialized: now 1786155750352532 us; error 30 us; skew 500 ppm
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:30.355347 32298 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:30.355790 32298 webserver.cc:460] Webserver started at http://127.30.160.3:46409/ using document root <none> and password file <none>
I20260808 02:22:30.356070 32298 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:30.356139 32298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:30.356297 32298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:30.357393 32298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "08ef3580db9242cb885995c9c9757dac"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "2121582bcb688ae393cdb1433583ee7c"
server_key_iv: "613031aa0fb76b46349e13d0adf36785"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.357793 32298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "08ef3580db9242cb885995c9c9757dac"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "2121582bcb688ae393cdb1433583ee7c"
server_key_iv: "613031aa0fb76b46349e13d0adf36785"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.359215 32298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:30.360038 32315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.360232 32298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:30.360355 32298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "08ef3580db9242cb885995c9c9757dac"
format_stamp: "Formatted at 2026-08-08 02:22:30 on dist-test-slave-zkpd"
server_key: "2121582bcb688ae393cdb1433583ee7c"
server_key_iv: "613031aa0fb76b46349e13d0adf36785"
server_key_version: "encryptionkey@0"
I20260808 02:22:30.360436 32298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:30.380045 32298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:30.382092 32298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:30.382272 32298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:30.382526 32298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:30.382921 32298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:30.383003 32298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.383047 32298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:30.383071 32298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:30.389184 32298 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:41659
I20260808 02:22:30.389230 32432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:41659 every 8 connection(s)
I20260808 02:22:30.389619 32298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:30.393254 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 32298
I20260808 02:22:30.393345 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:30.393535 31360 external_mini_cluster.cc:1428] Setting key 0b0b7201e142a0c9b9e79b691fa9c456
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:30.399691 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.390415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:51881 (local address 127.30.160.62:40901)
0808 02:22:30.390501 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.390504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.390676 (+ 172us) server_negotiation.cc:407] Connection header received
0808 02:22:30.391620 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.391621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.391632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.391689 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.392423 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.392935 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.394211 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.394510 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.394578 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.396690 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.396692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.396692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.396699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.398127 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.398525 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.398526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.398526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.398651 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.398909 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.398910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.398910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.399023 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.399077 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.399502 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.399600 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:22:30.399935 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.390364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40901 (local address 127.30.160.3:51881)
0808 02:22:30.390539 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.390578 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.391504 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.391766 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.391770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.392120 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:30.392323 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.392329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.393020 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.393021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.394135 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.394138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.394597 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.394598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.394711 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.395166 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:30.395175 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:30.396611 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:30.398195 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:30.398200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:30.398202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:30.398419 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:30.398731 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:30.398732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:30.398732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:30.398846 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:30.399204 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:30.399211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:30.399341 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.399457 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.399641 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260808 02:22:30.400403 32433 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40901
I20260808 02:22:30.400512 32433 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:30.400693 32433 heartbeater.cc:511] Master 127.30.160.62:40901 requested a full tablet report, sending...
I20260808 02:22:30.401079 31954 ts_manager.cc:194] Registered new tserver with Master: 08ef3580db9242cb885995c9c9757dac (127.30.160.3:41659)
I20260808 02:22:30.401464 31954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:51881
I20260808 02:22:30.405153 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:22:30.412629 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.406779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.30.160.62:40901)
0808 02:22:30.406839 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.406841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.406905 (+ 64us) server_negotiation.cc:407] Connection header received
0808 02:22:30.406955 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.406955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.406963 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.407014 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.407523 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.408037 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.408720 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.408899 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.408950 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.410167 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.410168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.410168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.410174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.411547 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.411934 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.411935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.411937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.411994 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.412183 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.412184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.412184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.412328 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.412406 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.412511 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.412567 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:22:30.414772 31954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:30.416478 31954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:30.423259 32447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.419078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:40307 (local address 127.0.0.1:48564)
0808 02:22:30.419232 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.419243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.419283 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.419795 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.419796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.419798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:30.419848 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.419849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.421145 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.421146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.421997 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.422000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.422918 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.422919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.423020 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.423035 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.423123 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.423175 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260808 02:22:30.423399 32449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.419266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:41659 (local address 127.0.0.1:45538)
0808 02:22:30.419406 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.419415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.419460 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.419800 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.419801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.419803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:30.419851 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.419853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.420867 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.420868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.422269 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.422273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.423154 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.423155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.423218 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.423224 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.423283 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.423326 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":54,"thread_start_us":87,"threads_started":1}
I20260808 02:22:30.423415 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.419447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.30.160.3:41659)
0808 02:22:30.419626 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.419630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.419639 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:30.419679 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.419681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.419695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.419967 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:30.420050 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.420760 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.422365 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.423059 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.423153 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.423208 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.423297 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.423339 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260808 02:22:30.423601 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.418854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33457 (local address 127.0.0.1:42446)
0808 02:22:30.418996 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.419028 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.419107 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.419475 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.419478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.419481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:30.419551 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.419553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.420979 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.420980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.422271 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.422273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.423387 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.423388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.423459 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.423463 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.423494 (+ 31us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.423533 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":30,"thread_start_us":74,"threads_started":1}
I20260808 02:22:30.423846 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.419222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.30.160.1:40307)
0808 02:22:30.419623 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.419628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.419638 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:30.419679 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.419681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.419696 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.419790 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:30.419962 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.421043 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.422105 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.422787 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.422875 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.422935 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.423692 (+ 757us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.423748 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":113,"threads_started":1}
I20260808 02:22:30.424044 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.419003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.30.160.2:33457)
0808 02:22:30.419220 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.419235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.419246 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:30.419292 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.419294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.419313 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.419440 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:30.419635 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.420874 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.422478 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.423248 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.423762 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.423840 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.423863 (+ 23us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.423963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260808 02:22:30.424439 32363 tablet_service.cc:1467] Processing CreateTablet for tablet 0980084a5d7644b6b3ad602e7ea0b1a9 (DEFAULT_TABLE table=test-table [id=5c1c256dbe5641ca8b9cc7896b93c6d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:30.424511 32081 tablet_service.cc:1467] Processing CreateTablet for tablet 0980084a5d7644b6b3ad602e7ea0b1a9 (DEFAULT_TABLE table=test-table [id=5c1c256dbe5641ca8b9cc7896b93c6d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:30.424727 32224 tablet_service.cc:1467] Processing CreateTablet for tablet 0980084a5d7644b6b3ad602e7ea0b1a9 (DEFAULT_TABLE table=test-table [id=5c1c256dbe5641ca8b9cc7896b93c6d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:30.424830 32363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0980084a5d7644b6b3ad602e7ea0b1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:30.424870 32081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0980084a5d7644b6b3ad602e7ea0b1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:30.425031 32224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0980084a5d7644b6b3ad602e7ea0b1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:30.427974 32451 tablet_bootstrap.cc:492] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: Bootstrap starting.
I20260808 02:22:30.428723 32451 tablet_bootstrap.cc:654] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:30.428759 32453 tablet_bootstrap.cc:492] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: Bootstrap starting.
I20260808 02:22:30.429359 32453 tablet_bootstrap.cc:654] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:30.429363 32451 log.cc:824] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:30.429411 32452 tablet_bootstrap.cc:492] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: Bootstrap starting.
I20260808 02:22:30.429981 32453 log.cc:824] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:30.430083 32452 tablet_bootstrap.cc:654] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:30.430254 32451 tablet_bootstrap.cc:492] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: No bootstrap required, opened a new log
I20260808 02:22:30.430339 32451 ts_tablet_manager.cc:1397] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:22:30.430656 32453 tablet_bootstrap.cc:492] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: No bootstrap required, opened a new log
I20260808 02:22:30.430739 32453 ts_tablet_manager.cc:1397] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:30.431002 32452 log.cc:824] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:30.432313 32453 raft_consensus.cc:348] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.432276 32451 raft_consensus.cc:348] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.432475 32452 tablet_bootstrap.cc:492] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: No bootstrap required, opened a new log
I20260808 02:22:30.432471 32453 raft_consensus.cc:374] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:30.432471 32451 raft_consensus.cc:374] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:30.432514 32453 raft_consensus.cc:729] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ef3580db9242cb885995c9c9757dac, State: Initialized, Role: FOLLOWER
I20260808 02:22:30.432514 32451 raft_consensus.cc:729] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe5f5870bb2411fb2fc5d034ad6b3ce, State: Initialized, Role: FOLLOWER
I20260808 02:22:30.432539 32452 ts_tablet_manager.cc:1397] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:22:30.432647 32453 consensus_queue.cc:260] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.432652 32451 consensus_queue.cc:260] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.432947 32453 ts_tablet_manager.cc:1428] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:30.432966 32451 ts_tablet_manager.cc:1428] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:22:30.432957 32433 heartbeater.cc:503] Master 127.30.160.62:40901 was elected leader, sending a full tablet report...
I20260808 02:22:30.433079 32152 heartbeater.cc:503] Master 127.30.160.62:40901 was elected leader, sending a full tablet report...
I20260808 02:22:30.434587 32452 raft_consensus.cc:348] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.434710 32452 raft_consensus.cc:374] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:30.434729 32452 raft_consensus.cc:729] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4795f882fa4c99a39c048bf5a2e479, State: Initialized, Role: FOLLOWER
I20260808 02:22:30.434860 32452 consensus_queue.cc:260] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.435117 32452 ts_tablet_manager.cc:1428] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:22:30.435137 32292 heartbeater.cc:503] Master 127.30.160.62:40901 was elected leader, sending a full tablet report...
W20260808 02:22:30.474011 32293 tablet.cc:2369] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:22:30.565587 32153 tablet.cc:2369] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:30.578513 32458 raft_consensus.cc:482] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:30.578650 32458 raft_consensus.cc:504] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.579100 32458 leader_election.cc:288] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb4795f882fa4c99a39c048bf5a2e479 (127.30.160.2:33457), 0fe5f5870bb2411fb2fc5d034ad6b3ce (127.30.160.1:40307)
I20260808 02:22:30.582376 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.579432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:42479 (local address 127.30.160.2:33457)
0808 02:22:30.579489 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.579491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.579504 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:22:30.579539 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.579540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.579551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.579625 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:30.579828 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.580612 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.581571 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.581968 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.582079 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.582089 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.582264 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.582308 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:22:30.582376 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.579300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33457 (local address 127.30.160.3:42479)
0808 02:22:30.579376 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.579389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.579479 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.579678 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.579679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.579681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:30.579748 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.579749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.580696 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.580696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.581475 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.581477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.582088 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.582089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.582162 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.582166 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.582253 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.582297 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:22:30.582469 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.579530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:40153 (local address 127.30.160.1:40307)
0808 02:22:30.579575 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.579577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.579701 (+ 124us) server_negotiation.cc:407] Connection header received
0808 02:22:30.579774 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.579775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.579789 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.579843 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:30.580030 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.580718 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.581719 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.582092 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.582183 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.582191 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.582360 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.582403 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:30.582480 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.579433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:40307 (local address 127.30.160.3:40153)
0808 02:22:30.579617 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:30.579634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:30.579708 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:30.579861 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:30.579862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:30.579863 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:30.579955 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.579956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.580839 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.580840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.581615 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:30.581617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.582183 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.582184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:30.582275 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.582278 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:30.582353 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:22:30.582396 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":121,"threads_started":1}
I20260808 02:22:30.582846 32102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0980084a5d7644b6b3ad602e7ea0b1a9" candidate_uuid: "08ef3580db9242cb885995c9c9757dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" is_pre_election: true
I20260808 02:22:30.582852 32245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0980084a5d7644b6b3ad602e7ea0b1a9" candidate_uuid: "08ef3580db9242cb885995c9c9757dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4795f882fa4c99a39c048bf5a2e479" is_pre_election: true
I20260808 02:22:30.583021 32102 raft_consensus.cc:2454] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ef3580db9242cb885995c9c9757dac in term 0.
I20260808 02:22:30.583050 32245 raft_consensus.cc:2454] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ef3580db9242cb885995c9c9757dac in term 0.
I20260808 02:22:30.583272 32319 leader_election.cc:302] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ef3580db9242cb885995c9c9757dac, 0fe5f5870bb2411fb2fc5d034ad6b3ce; no voters:
I20260808 02:22:30.583397 32458 raft_consensus.cc:2781] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:30.583458 32458 raft_consensus.cc:482] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:30.583489 32458 raft_consensus.cc:3037] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:30.584282 32458 raft_consensus.cc:504] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.584419 32458 leader_election.cc:288] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [CANDIDATE]: Term 1 election: Requested vote from peers cb4795f882fa4c99a39c048bf5a2e479 (127.30.160.2:33457), 0fe5f5870bb2411fb2fc5d034ad6b3ce (127.30.160.1:40307)
I20260808 02:22:30.584640 32245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0980084a5d7644b6b3ad602e7ea0b1a9" candidate_uuid: "08ef3580db9242cb885995c9c9757dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4795f882fa4c99a39c048bf5a2e479"
I20260808 02:22:30.584659 32102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0980084a5d7644b6b3ad602e7ea0b1a9" candidate_uuid: "08ef3580db9242cb885995c9c9757dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce"
I20260808 02:22:30.584754 32245 raft_consensus.cc:3037] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:30.584767 32102 raft_consensus.cc:3037] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:30.585759 32102 raft_consensus.cc:2454] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 0fe5f5870bb2411fb2fc5d034ad6b3ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ef3580db9242cb885995c9c9757dac in term 1.
I20260808 02:22:30.585768 32245 raft_consensus.cc:2454] T 0980084a5d7644b6b3ad602e7ea0b1a9 P cb4795f882fa4c99a39c048bf5a2e479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ef3580db9242cb885995c9c9757dac in term 1.
I20260808 02:22:30.585951 32319 leader_election.cc:302] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ef3580db9242cb885995c9c9757dac, 0fe5f5870bb2411fb2fc5d034ad6b3ce; no voters:
I20260808 02:22:30.586057 32458 raft_consensus.cc:2781] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:30.586228 32458 raft_consensus.cc:686] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [term 1 LEADER]: Becoming Leader. State: Replica: 08ef3580db9242cb885995c9c9757dac, State: Running, Role: LEADER
I20260808 02:22:30.586330 32458 consensus_queue.cc:237] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } }
I20260808 02:22:30.587193 31954 catalog_manager.cc:5495] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ef3580db9242cb885995c9c9757dac (127.30.160.3). New cstate: current_term: 1 leader_uuid: "08ef3580db9242cb885995c9c9757dac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4795f882fa4c99a39c048bf5a2e479" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe5f5870bb2411fb2fc5d034ad6b3ce" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 40307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08ef3580db9242cb885995c9c9757dac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41659 } health_report { overall_health: HEALTHY } } }
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:30.611724 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.604356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.30.160.1:40307)
0808 02:22:30.604451 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.604453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.604510 (+ 57us) server_negotiation.cc:407] Connection header received
0808 02:22:30.604558 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.604559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.604568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.604617 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.605195 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.605747 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.606439 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.606639 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.606689 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.609121 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.609133 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.609133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.609139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.610669 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.610984 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.610987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.610988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.611054 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.611227 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.611228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.611228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.611370 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.611474 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.611536 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.611601 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:30.636106 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.629203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.30.160.1:40307)
0808 02:22:30.629274 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.629276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.629319 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:22:30.629381 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.629382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.629391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.629438 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.630021 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.630498 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.631212 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.631409 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.631513 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.633767 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.633773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.633773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.633779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.634978 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.635287 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.635287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.635288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.635348 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.635564 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.635565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.635565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.635652 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.635723 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.635973 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.636039 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 02:22:30.640249 32434 tablet.cc:2369] T 0980084a5d7644b6b3ad602e7ea0b1a9 P 08ef3580db9242cb885995c9c9757dac: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[31905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:30.659636 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:30.653016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.30.160.1:40307)
0808 02:22:30.653107 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:30.653109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:30.653168 (+ 59us) server_negotiation.cc:407] Connection header received
0808 02:22:30.653217 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:30.653218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:30.653227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:30.653277 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:30.653930 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.654402 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.655100 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:30.655273 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:30.655322 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:30.657510 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:30.657512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:30.657512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:30.657518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:30.658693 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.659026 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.659027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.659027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.659078 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:30.659240 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:30.659240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:30.659240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:30.659339 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:30.659413 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:30.659511 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:22:30.659568 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 02:22:30.660248 32061 tablet_service.cc:2908] Rejecting scan request for tablet 0980084a5d7644b6b3ad602e7ea0b1a9: Uninitialized: clean time has not yet been initialized
I20260808 02:22:30.661158 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 32016
I20260808 02:22:30.667856 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 32158
I20260808 02:22:30.675134 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 32298
I20260808 02:22:30.680837 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 31919
2026-08-08T02:22:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2139 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[32476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[32476](info): set up 2 sockets
Aug 08 02:22:30 dist-test-slave-zkpd krb5kdc[32476](info): commencing operation
krb5kdc: starting...
W20260808 02:22:31.755789 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:31 dist-test-slave-zkpd krb5kdc[32476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:31.878571 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37631
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:37631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:31.960456 32489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:31.960701 32489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:31.960736 32489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:31.961985 32489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:31.962023 32489 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:22:31.962038 32489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:31.962054 32489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:31.962066 32489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:31.962080 32489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:31.963692 32489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:37631
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:31.963994 32489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:31.964214 32489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:31.969588 32495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:31.969594 32494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:31.969575 32497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:31.969770 32489 server_base.cc:1034] running on GCE node
I20260808 02:22:31.970085 32489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:31.970429 32489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:31.971601 32489 hybrid_clock.cc:648] HybridClock initialized: now 1786155751971588 us; error 43 us; skew 500 ppm
Aug 08 02:22:31 dist-test-slave-zkpd krb5kdc[32476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:31.974280 32489 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:31.974740 32489 webserver.cc:460] Webserver started at http://127.30.160.62:39973/ using document root <none> and password file <none>
I20260808 02:22:31.974928 32489 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:31.974968 32489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:31.975060 32489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:31.976150 32489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "b92d28138f7648af8e6d1e4e7a5e639b"
format_stamp: "Formatted at 2026-08-08 02:22:31 on dist-test-slave-zkpd"
server_key: "4ab1877c64d20508e92646d79ba4d42a"
server_key_iv: "e8a0824fd7a0e93970517f8800e5a2bc"
server_key_version: "encryptionkey@0"
I20260808 02:22:31.976462 32489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "b92d28138f7648af8e6d1e4e7a5e639b"
format_stamp: "Formatted at 2026-08-08 02:22:31 on dist-test-slave-zkpd"
server_key: "4ab1877c64d20508e92646d79ba4d42a"
server_key_iv: "e8a0824fd7a0e93970517f8800e5a2bc"
server_key_version: "encryptionkey@0"
I20260808 02:22:31.977734 32489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:31.978436 32504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:31.978631 32489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:31.978719 32489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "b92d28138f7648af8e6d1e4e7a5e639b"
format_stamp: "Formatted at 2026-08-08 02:22:31 on dist-test-slave-zkpd"
server_key: "4ab1877c64d20508e92646d79ba4d42a"
server_key_iv: "e8a0824fd7a0e93970517f8800e5a2bc"
server_key_version: "encryptionkey@0"
I20260808 02:22:31.978773 32489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:31.987679 32489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:31.989655 32489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:31.989765 32489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:31.994508 32489 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:37631
I20260808 02:22:31.994596 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:37631 every 8 connection(s)
I20260808 02:22:31.994921 32489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:31.995628 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:31.997952 32568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: Bootstrap starting.
I20260808 02:22:31.998561 32568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:31.998849 32568 log.cc:824] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:31.999667 32568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: No bootstrap required, opened a new log
I20260808 02:22:32.002228 32568 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37631 } }
I20260808 02:22:32.002409 32568 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.002462 32568 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92d28138f7648af8e6d1e4e7a5e639b, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.002674 32568 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37631 } }
I20260808 02:22:32.002789 32568 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:32.002840 32568 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:32.002899 32568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.003516 32568 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37631 } }
I20260808 02:22:32.003540 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 32489
I20260808 02:22:32.003654 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:32.003652 32568 leader_election.cc:302] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92d28138f7648af8e6d1e4e7a5e639b; no voters:
I20260808 02:22:32.003854 31360 external_mini_cluster.cc:1428] Setting key 609bad564ef82f22c30c6cfdb18efe00
I20260808 02:22:32.003871 32568 leader_election.cc:288] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:32.003923 32572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:32.004108 32572 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [term 1 LEADER]: Becoming Leader. State: Replica: b92d28138f7648af8e6d1e4e7a5e639b, State: Running, Role: LEADER
I20260808 02:22:32.004204 32568 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:32.004238 32572 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37631 } }
I20260808 02:22:32.004932 32574 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92d28138f7648af8e6d1e4e7a5e639b. Latest consensus state: current_term: 1 leader_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37631 } } }
I20260808 02:22:32.004951 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d28138f7648af8e6d1e4e7a5e639b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37631 } } }
I20260808 02:22:32.005064 32574 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:32.005079 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:32.005889 32584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:32.006501 32584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:32.008169 32584 catalog_manager.cc:1347] Generated new cluster ID: 233e898c0ab9432f8dacf7a25bf38c32
I20260808 02:22:32.008226 32584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:32.016013 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.004370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.30.160.62:37631)
0808 02:22:32.004566 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.004578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.004728 (+ 150us) server_negotiation.cc:407] Connection header received
0808 02:22:32.004883 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.004892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.004970 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.005114 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.006102 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.007471 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.008616 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.009218 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.009329 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.012087 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.012092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.012092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.012104 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.014188 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.014626 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.014629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.014630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.014756 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.014991 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.014991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.014992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.015273 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.015425 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.015555 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.015684 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260808 02:22:32.024420 32584 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:32.025187 32584 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:32.029160 32584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: Generated new TSK 0
I20260808 02:22:32.029349 32584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:32.059244 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37631
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:32.142326 32597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:32.142520 32597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:32.142544 32597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:32.143700 32597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:32.143745 32597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:32.143765 32597 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:32.143811 32597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:32.145287 32597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:32.145550 32597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:32.145787 32597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:32.151347 32605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:32.151458 32597 server_base.cc:1034] running on GCE node
W20260808 02:22:32.151404 32603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:32.151362 32602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:32.151788 32597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:32.152091 32597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:32.153251 32597 hybrid_clock.cc:648] HybridClock initialized: now 1786155752153231 us; error 38 us; skew 500 ppm
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:32.155763 32597 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:32.156275 32597 webserver.cc:460] Webserver started at http://127.30.160.1:35123/ using document root <none> and password file <none>
I20260808 02:22:32.156525 32597 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:32.156594 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:32.156750 32597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:32.157807 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "b7d435d3b4b84dd497f804240c6afcc6"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "5c028679b157bcaf3785f633ecd28538"
server_key_iv: "990b68c3541159c3517d00c5b8218b68"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.158166 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "b7d435d3b4b84dd497f804240c6afcc6"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "5c028679b157bcaf3785f633ecd28538"
server_key_iv: "990b68c3541159c3517d00c5b8218b68"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.159473 32597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:32.160310 32612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.160483 32597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:32.160588 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "b7d435d3b4b84dd497f804240c6afcc6"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "5c028679b157bcaf3785f633ecd28538"
server_key_iv: "990b68c3541159c3517d00c5b8218b68"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.160661 32597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:32.177644 32597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:32.179411 32597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:32.179569 32597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:32.180195 32597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:32.180254 32597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.180305 32597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:32.180333 32597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:32.187551 32597 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42739
I20260808 02:22:32.187633 32730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42739 every 8 connection(s)
I20260808 02:22:32.187989 32597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:32.190567 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.180657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:55579 (local address 127.30.160.62:37631)
0808 02:22:32.180722 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.180724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.180947 (+ 223us) server_negotiation.cc:407] Connection header received
0808 02:22:32.181516 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.181518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.181535 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.181598 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.182462 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.183028 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.184707 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.185085 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.185186 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.187514 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.187515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.187516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.187524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.189133 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.189554 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.189555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.189556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.189621 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.189891 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.189892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.189893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.190012 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.190084 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.190385 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.190475 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:22:32.190743 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.180648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.30.160.1:55579)
0808 02:22:32.180826 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.180861 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.181422 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.181666 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.181669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.182054 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.182376 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.182382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.183134 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.183135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.184599 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.184603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.185226 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.185227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.185360 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.185903 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.185911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.187417 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.189234 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.189242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.189244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.189475 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.189691 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.189692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.189692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.189813 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.190087 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.190095 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.190231 (+ 136us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.190358 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.190519 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260808 02:22:32.191243 32731 heartbeater.cc:347] Connected to a master server at 127.30.160.62:37631
I20260808 02:22:32.191334 32731 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:32.191521 32731 heartbeater.cc:511] Master 127.30.160.62:37631 requested a full tablet report, sending...
I20260808 02:22:32.192158 32520 ts_manager.cc:194] Registered new tserver with Master: b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739)
I20260808 02:22:32.192889 32520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:55579
I20260808 02:22:32.194664 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 32597
I20260808 02:22:32.194787 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:32.195015 31360 external_mini_cluster.cc:1428] Setting key 7628ac539b7d96851dafdc19c6f8af12
I20260808 02:22:32.198177 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.192385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:33979 (local address 127.30.160.62:37631)
0808 02:22:32.192422 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.192423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.192559 (+ 136us) server_negotiation.cc:407] Connection header received
0808 02:22:32.192632 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.192634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.192643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.192687 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.193055 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.193735 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.194432 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.194611 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.194691 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.195686 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.195688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.195688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.195695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.197036 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.197394 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.197395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.197396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.197451 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.197666 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.197667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.197667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.197777 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.197850 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.198013 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.198090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 02:22:32.198194 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.192346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.30.160.1:33979)
0808 02:22:32.192479 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.192490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.192574 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.192701 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.192702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.192890 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.192937 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.192939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.193855 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.193856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.194362 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.194363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.194691 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.194692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.194773 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.195249 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.195255 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.195572 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.197123 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.197125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.197125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.197304 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.197534 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.197535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.197536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.197601 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.197855 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.197856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.197903 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.197988 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.198091 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:32.220057 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37631
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:32.304006 32740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:32.304286 32740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:32.304315 32740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:32.305642 32740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:32.305686 32740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:32.305707 32740 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:32.305724 32740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:32.307165 32740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:32.307351 32740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:32.307530 32740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:32.312676 32745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:32.312654 32748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:32.312654 32746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:32.312747 32740 server_base.cc:1034] running on GCE node
I20260808 02:22:32.313078 32740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:32.313369 32740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:32.314529 32740 hybrid_clock.cc:648] HybridClock initialized: now 1786155752314513 us; error 28 us; skew 500 ppm
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:32.317071 32740 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:32.317512 32740 webserver.cc:460] Webserver started at http://127.30.160.2:42359/ using document root <none> and password file <none>
I20260808 02:22:32.317740 32740 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:32.317809 32740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:32.317970 32740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:32.319010 32740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "73d11b9ebab54dbe8486548d752d2c30"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "743654971db011a9d94372b9c11a4be4"
server_key_iv: "a38664ec1af73464319ad649a3569042"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.319380 32740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "73d11b9ebab54dbe8486548d752d2c30"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "743654971db011a9d94372b9c11a4be4"
server_key_iv: "a38664ec1af73464319ad649a3569042"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.320807 32740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:32.321563 32755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.321740 32740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:32.321851 32740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "73d11b9ebab54dbe8486548d752d2c30"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "743654971db011a9d94372b9c11a4be4"
server_key_iv: "a38664ec1af73464319ad649a3569042"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.321925 32740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:32.334164 32740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:32.336144 32740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:32.336303 32740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:32.336809 32740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:32.336871 32740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.336936 32740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:32.336966 32740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.344190 32740 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:40687
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:32.344278 406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:40687 every 8 connection(s)
I20260808 02:22:32.344673 32740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:32.347551 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.337379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:44515 (local address 127.30.160.62:37631)
0808 02:22:32.337420 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.337422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.337648 (+ 226us) server_negotiation.cc:407] Connection header received
0808 02:22:32.338185 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.338187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.338201 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.338282 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.339254 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.340096 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.341750 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.342067 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.342157 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.344747 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.344749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.344749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.344756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.346157 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.346585 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.346587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.346587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.346649 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.346911 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.346912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.346912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.347029 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.347098 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.347379 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.347457 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:22:32.347747 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.337367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.30.160.2:44515)
0808 02:22:32.337523 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.337558 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.338074 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.338401 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.338405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.338825 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.339133 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.339140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.340189 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.340190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.341634 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.341637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.342175 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.342176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.342328 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.342983 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.342993 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.344633 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.346261 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.346269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.346271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.346493 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.346723 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.346724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.346724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.346837 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.347103 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.347110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.347245 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.347354 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.347513 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260808 02:22:32.348348 407 heartbeater.cc:347] Connected to a master server at 127.30.160.62:37631
I20260808 02:22:32.348434 407 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:32.348631 407 heartbeater.cc:511] Master 127.30.160.62:37631 requested a full tablet report, sending...
I20260808 02:22:32.349010 32520 ts_manager.cc:194] Registered new tserver with Master: 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2:40687)
I20260808 02:22:32.349387 32520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:44515
I20260808 02:22:32.354271 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.349464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:43433 (local address 127.30.160.62:37631)
0808 02:22:32.349506 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.349508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.349601 (+ 93us) server_negotiation.cc:407] Connection header received
0808 02:22:32.349657 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.349658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.349668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.349721 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.350068 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.350504 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.351137 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.351310 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.351359 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.352255 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.352255 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.352255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.352260 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.353411 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.353706 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.353707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.353707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.353755 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.353901 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.353901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.353901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.354015 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.354062 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.354162 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.354215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:32.354286 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.349398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.30.160.2:43433)
0808 02:22:32.349541 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.349558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.349611 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.349738 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.349738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.349934 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.349994 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.349996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.350588 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.350589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.351065 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.351066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.351374 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.351374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.351427 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.351861 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.351866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.352187 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.353480 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.353480 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.353481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.353636 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.353810 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.353811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.353811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.353861 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.354064 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.354065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.354099 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.354148 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.354220 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260808 02:22:32.354816 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 32740
I20260808 02:22:32.354892 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:32.355115 31360 external_mini_cluster.cc:1428] Setting key 5e1c7ebd379a3b83f3695893eb3061ce
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:32.380228 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37631
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:32.463505 416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:32.463748 416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:32.463789 416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:32.464977 416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:32.465044 416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:32.465080 416 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:32.465121 416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:32.466542 416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:37631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:32.466812 416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:32.467052 416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:32.472025 422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:32.472034 425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:32.472040 423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:32.472389 416 server_base.cc:1034] running on GCE node
I20260808 02:22:32.472604 416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:32.472846 416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:32.474013 416 hybrid_clock.cc:648] HybridClock initialized: now 1786155752474001 us; error 42 us; skew 500 ppm
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:32.476630 416 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:32.477111 416 webserver.cc:460] Webserver started at http://127.30.160.3:45809/ using document root <none> and password file <none>
I20260808 02:22:32.477360 416 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:32.477428 416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:32.477578 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:32.478638 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "ea1d5653f326471496ac8d315b1d1360"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "6f9e31bcafa7da122d5d207ff3386472"
server_key_iv: "c527c2ebb7bdf3069645d928afcc8a82"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.479013 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "ea1d5653f326471496ac8d315b1d1360"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "6f9e31bcafa7da122d5d207ff3386472"
server_key_iv: "c527c2ebb7bdf3069645d928afcc8a82"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.480459 416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:32.481245 433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.481520 416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:32.481637 416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "ea1d5653f326471496ac8d315b1d1360"
format_stamp: "Formatted at 2026-08-08 02:22:32 on dist-test-slave-zkpd"
server_key: "6f9e31bcafa7da122d5d207ff3386472"
server_key_iv: "c527c2ebb7bdf3069645d928afcc8a82"
server_key_version: "encryptionkey@0"
I20260808 02:22:32.481712 416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:32.491995 416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:32.494269 416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:32.494429 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:32.495028 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:32.495095 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.495123 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:32.495139 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:32.502802 416 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:41233
I20260808 02:22:32.502878 553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:41233 every 8 connection(s)
I20260808 02:22:32.503170 416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:32.504385 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.495382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:56657 (local address 127.30.160.62:37631)
0808 02:22:32.495429 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.495431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.495624 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:22:32.496218 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.496219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.496235 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.496311 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.496938 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.497442 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.498453 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.498659 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.498736 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.501097 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.501099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.501099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.501108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.502862 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.503304 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.503305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.503306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.503369 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.503655 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.503657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.503657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.503803 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.503887 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.504186 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.504286 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:22:32.504508 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.495356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.30.160.3:56657)
0808 02:22:32.495516 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.495549 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.496108 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.496380 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.496383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.496607 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.496806 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.496835 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.497552 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.497553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.498348 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.498350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.498744 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.498745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.498915 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.499424 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.499435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.501000 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.502932 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.502937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.502939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.503197 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.503446 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.503447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.503447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.503577 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.503877 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.503882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.504058 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.504153 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.504297 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260808 02:22:32.505050 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 416
I20260808 02:22:32.505132 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:32.505376 31360 external_mini_cluster.cc:1428] Setting key 45b41b96858df03807770a55d9124e58
I20260808 02:22:32.505530 554 heartbeater.cc:347] Connected to a master server at 127.30.160.62:37631
I20260808 02:22:32.505617 554 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:32.505808 554 heartbeater.cc:511] Master 127.30.160.62:37631 requested a full tablet report, sending...
I20260808 02:22:32.506189 32521 ts_manager.cc:194] Registered new tserver with Master: ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233)
I20260808 02:22:32.506556 32521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:56657
I20260808 02:22:32.507153 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:22:32.512934 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.506074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.30.160.3:58839)
0808 02:22:32.506799 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.506810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.507315 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.507473 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.507474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.507649 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.507694 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.507695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.508407 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.508408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.508923 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.508924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.509400 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.509401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.509463 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.509875 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.509879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.510128 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.511884 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.511885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.511885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.512172 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.512394 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.512394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.512395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.512446 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.512708 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.512709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.512744 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.512790 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.512873 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":63,"threads_started":1}
I20260808 02:22:32.513314 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.506136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:58839 (local address 127.30.160.62:37631)
0808 02:22:32.506177 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.506179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.506858 (+ 679us) server_negotiation.cc:407] Connection header received
0808 02:22:32.507413 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.507414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.507424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.507470 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.507790 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.508315 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.509008 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.509318 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.509387 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.510213 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.510214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.510215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.510223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.511801 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.512255 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.512257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.512257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.512320 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.512507 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.512508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.512508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.512642 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.512714 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.512829 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.512903 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:32.531036 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.524515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.30.160.62:37631)
0808 02:22:32.524571 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.524572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.524592 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:22:32.524613 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.524613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.524622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.524709 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.525240 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.525693 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.526410 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.526585 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.526636 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.528742 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.528744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.528744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.528751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.530128 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.530442 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.530443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.530443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.530491 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.530653 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.530653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.530653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.530745 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.530828 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.530917 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.530972 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:22:32.532783 32521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:32.534294 32521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:32.541618 574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.536667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:40687 (local address 127.0.0.1:44478)
0808 02:22:32.537056 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.537100 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.537179 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.537492 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.537495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.537497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:32.537563 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.537565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.538889 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.538889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.540258 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.540260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.541265 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.541266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.541378 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.541395 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.541480 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.541539 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":31,"thread_start_us":67,"threads_started":1}
I20260808 02:22:32.541682 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.536858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.30.160.2:40687)
0808 02:22:32.537018 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.537024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.537176 (+ 152us) server_negotiation.cc:407] Connection header received
0808 02:22:32.537273 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.537293 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.537312 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.537398 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:32.537665 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.538801 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.540368 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.541161 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.541238 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.541302 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.541513 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.541580 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260808 02:22:32.542373 573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.536674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42739 (local address 127.0.0.1:38476)
0808 02:22:32.537438 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.537447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.537490 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.537741 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.537742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.537746 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:32.537812 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.537813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.540045 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.540046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.541112 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.541115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.542146 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.542147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.542205 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.542210 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.542257 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.542296 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":22,"thread_start_us":110,"threads_started":1}
I20260808 02:22:32.542520 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.536668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:41233 (local address 127.0.0.1:52686)
0808 02:22:32.537279 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.537287 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.537354 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.537741 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.537742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.537744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:32.537816 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.537817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.539036 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.539037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.540257 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.540260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.542256 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.542257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.542324 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.542329 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.542385 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.542426 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
I20260808 02:22:32.542418 338 tablet_service.cc:1467] Processing CreateTablet for tablet 45791ee52ade42c3964f64a5b870c5f6 (DEFAULT_TABLE table=test-table [id=49d2b26e11604ef289c26dfd3db97660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:32.542685 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.536980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.30.160.1:42739)
0808 02:22:32.537143 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.537147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.537544 (+ 397us) server_negotiation.cc:407] Connection header received
0808 02:22:32.537589 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.537592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.537607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.537675 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:32.539035 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.539955 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.541235 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.541995 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.542061 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.542111 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.542539 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.542584 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260808 02:22:32.542835 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45791ee52ade42c3964f64a5b870c5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.542941 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.536852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.30.160.3:41233)
0808 02:22:32.537072 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.537077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.537356 (+ 279us) server_negotiation.cc:407] Connection header received
0808 02:22:32.537506 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.537509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.537594 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.537670 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:32.537900 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.538944 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.541337 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.542098 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.542166 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.542222 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.542401 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.542829 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
I20260808 02:22:32.543362 32662 tablet_service.cc:1467] Processing CreateTablet for tablet 45791ee52ade42c3964f64a5b870c5f6 (DEFAULT_TABLE table=test-table [id=49d2b26e11604ef289c26dfd3db97660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:32.543557 485 tablet_service.cc:1467] Processing CreateTablet for tablet 45791ee52ade42c3964f64a5b870c5f6 (DEFAULT_TABLE table=test-table [id=49d2b26e11604ef289c26dfd3db97660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:32.543704 32662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45791ee52ade42c3964f64a5b870c5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.543845 485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45791ee52ade42c3964f64a5b870c5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.546030 578 tablet_bootstrap.cc:492] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Bootstrap starting.
I20260808 02:22:32.546624 578 tablet_bootstrap.cc:654] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.546869 578 log.cc:824] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:32.547418 580 tablet_bootstrap.cc:492] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Bootstrap starting.
I20260808 02:22:32.547555 581 tablet_bootstrap.cc:492] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Bootstrap starting.
I20260808 02:22:32.547618 578 tablet_bootstrap.cc:492] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: No bootstrap required, opened a new log
I20260808 02:22:32.547685 578 ts_tablet_manager.cc:1397] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:32.548166 581 tablet_bootstrap.cc:654] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.548447 580 tablet_bootstrap.cc:654] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.548539 581 log.cc:824] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:32.548789 580 log.cc:824] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:32.549309 581 tablet_bootstrap.cc:492] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: No bootstrap required, opened a new log
I20260808 02:22:32.549438 581 ts_tablet_manager.cc:1397] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:22:32.549611 580 tablet_bootstrap.cc:492] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: No bootstrap required, opened a new log
I20260808 02:22:32.549679 580 ts_tablet_manager.cc:1397] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:22:32.550050 578 raft_consensus.cc:348] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.550208 578 raft_consensus.cc:374] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.550256 578 raft_consensus.cc:729] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d11b9ebab54dbe8486548d752d2c30, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.550386 578 consensus_queue.cc:260] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.550689 578 ts_tablet_manager.cc:1428] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:22:32.550696 407 heartbeater.cc:503] Master 127.30.160.62:37631 was elected leader, sending a full tablet report...
I20260808 02:22:32.551573 581 raft_consensus.cc:348] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.551633 580 raft_consensus.cc:348] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.551733 581 raft_consensus.cc:374] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.551757 580 raft_consensus.cc:374] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.551775 581 raft_consensus.cc:729] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d435d3b4b84dd497f804240c6afcc6, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.551791 580 raft_consensus.cc:729] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1d5653f326471496ac8d315b1d1360, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.551918 580 consensus_queue.cc:260] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.551930 581 consensus_queue.cc:260] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.552194 581 ts_tablet_manager.cc:1428] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:22:32.552194 580 ts_tablet_manager.cc:1428] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:32.552253 32731 heartbeater.cc:503] Master 127.30.160.62:37631 was elected leader, sending a full tablet report...
I20260808 02:22:32.552485 554 heartbeater.cc:503] Master 127.30.160.62:37631 was elected leader, sending a full tablet report...
I20260808 02:22:32.560631 584 raft_consensus.cc:482] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:32.560724 584 raft_consensus.cc:504] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.561105 584 leader_election.cc:288] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233), b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739)
I20260808 02:22:32.564535 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.561286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42739 (local address 127.30.160.2:38623)
0808 02:22:32.561361 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.561373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.561417 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.561632 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.561633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.561635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:32.561729 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.561731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.562665 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.562665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.563527 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.563530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.564221 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.564223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.564335 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.564338 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.564418 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.564460 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260808 02:22:32.564539 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.561369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:38623 (local address 127.30.160.1:42739)
0808 02:22:32.561410 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.561412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.561421 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:32.561519 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.561520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.561537 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.561618 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:32.561805 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.562575 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.563620 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.564139 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.564224 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.564234 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.564428 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.564481 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:32.564543 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.561360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:45401 (local address 127.30.160.3:41233)
0808 02:22:32.561417 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.561419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.561580 (+ 161us) server_negotiation.cc:407] Connection header received
0808 02:22:32.561627 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.561628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.561637 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.561697 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:32.561859 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.562578 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.563617 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.564138 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.564226 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.564234 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.564422 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.564471 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:22:32.564860 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45791ee52ade42c3964f64a5b870c5f6" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d435d3b4b84dd497f804240c6afcc6" is_pre_election: true
I20260808 02:22:32.564966 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.561291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:41233 (local address 127.30.160.2:45401)
0808 02:22:32.561507 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.561516 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.561570 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.561707 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.561708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.561709 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:32.561789 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.561791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.562721 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.562723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.563524 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.563527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.564225 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.564226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.564313 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.564317 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.564837 (+ 520us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.564883 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":18,"thread_start_us":109,"threads_started":1}
I20260808 02:22:32.565047 32683 raft_consensus.cc:2454] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 0.
I20260808 02:22:32.565320 32756 leader_election.cc:302] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d11b9ebab54dbe8486548d752d2c30, b7d435d3b4b84dd497f804240c6afcc6; no voters:
I20260808 02:22:32.565462 584 raft_consensus.cc:2781] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:32.565452 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45791ee52ade42c3964f64a5b870c5f6" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1d5653f326471496ac8d315b1d1360" is_pre_election: true
I20260808 02:22:32.565542 584 raft_consensus.cc:482] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:32.565568 584 raft_consensus.cc:3037] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.565601 506 raft_consensus.cc:2454] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 0.
I20260808 02:22:32.566596 584 raft_consensus.cc:504] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.566744 584 leader_election.cc:288] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 election: Requested vote from peers ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233), b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739)
I20260808 02:22:32.566906 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45791ee52ade42c3964f64a5b870c5f6" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1d5653f326471496ac8d315b1d1360"
I20260808 02:22:32.566987 506 raft_consensus.cc:3037] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.567685 506 raft_consensus.cc:2454] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 1.
I20260808 02:22:32.567816 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45791ee52ade42c3964f64a5b870c5f6" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d435d3b4b84dd497f804240c6afcc6"
I20260808 02:22:32.567860 32757 leader_election.cc:302] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d11b9ebab54dbe8486548d752d2c30, ea1d5653f326471496ac8d315b1d1360; no voters:
I20260808 02:22:32.567922 32683 raft_consensus.cc:3037] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.568014 584 raft_consensus.cc:2781] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:32.568181 584 raft_consensus.cc:686] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 LEADER]: Becoming Leader. State: Replica: 73d11b9ebab54dbe8486548d752d2c30, State: Running, Role: LEADER
I20260808 02:22:32.568280 584 consensus_queue.cc:237] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } }
I20260808 02:22:32.568871 32683 raft_consensus.cc:2454] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 1.
I20260808 02:22:32.569149 32521 catalog_manager.cc:5495] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "73d11b9ebab54dbe8486548d752d2c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:32.577350 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.574740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.30.160.2:40687)
0808 02:22:32.574785 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.574787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.574829 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:22:32.574916 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.574917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.574935 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.575017 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:32.575168 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.575619 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.576503 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.576697 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.576775 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.576996 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:32.577091 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:32.577180 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.577237 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.577259 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:22:32.578938 32683 raft_consensus.cc:1261] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Refusing update from remote peer 73d11b9ebab54dbe8486548d752d2c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:32.578965 506 raft_consensus.cc:1261] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Refusing update from remote peer 73d11b9ebab54dbe8486548d752d2c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:32.579260 584 consensus_queue.cc:1035] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:32.579432 588 consensus_queue.cc:1035] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:32.581864 591 mvcc.cc:205] Tried to move back new op lower bound from 7316093962560978944 to 7316093962520395776. Current Snapshot: MvccSnapshot[applied={T|T < 7316093962560978944}]
I20260808 02:22:32.582175 593 mvcc.cc:205] Tried to move back new op lower bound from 7316093962560978944 to 7316093962520395776. Current Snapshot: MvccSnapshot[applied={T|T < 7316093962560978944}]
I20260808 02:22:32.582938 592 mvcc.cc:205] Tried to move back new op lower bound from 7316093962560978944 to 7316093962520395776. Current Snapshot: MvccSnapshot[applied={T|T < 7316093962560978944}]
I20260808 02:22:32.585717 32520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:32.585860 32520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:32.586668 32520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: Sending DeleteTablet for 3 replicas of tablet 45791ee52ade42c3964f64a5b870c5f6
I20260808 02:22:32.587001 485 tablet_service.cc:1514] Processing DeleteTablet for tablet 45791ee52ade42c3964f64a5b870c5f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:32 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:52686
I20260808 02:22:32.587093 338 tablet_service.cc:1514] Processing DeleteTablet for tablet 45791ee52ade42c3964f64a5b870c5f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:32 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:44478
I20260808 02:22:32.587100 32662 tablet_service.cc:1514] Processing DeleteTablet for tablet 45791ee52ade42c3964f64a5b870c5f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:32 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:38476
I20260808 02:22:32.587332 603 tablet_replica.cc:330] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: stopping tablet replica
I20260808 02:22:32.587435 603 raft_consensus.cc:2229] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:32.587530 603 raft_consensus.cc:2258] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:32.587800 601 tablet_replica.cc:330] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: stopping tablet replica
I20260808 02:22:32.587776 32520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:32.587965 32520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:32.588056 603 ts_tablet_manager.cc:1905] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:32.588201 601 raft_consensus.cc:2229] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:32.589275 338 tablet_service.cc:1467] Processing CreateTablet for tablet a330d280ac284e3fa4ae9bd5baa3fa80 (DEFAULT_TABLE table=test-table [id=b8ec4dfedd384e43908c8b7c4fb78ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:32.589421 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a330d280ac284e3fa4ae9bd5baa3fa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.589586 485 tablet_service.cc:1467] Processing CreateTablet for tablet a330d280ac284e3fa4ae9bd5baa3fa80 (DEFAULT_TABLE table=test-table [id=b8ec4dfedd384e43908c8b7c4fb78ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:32.589701 485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a330d280ac284e3fa4ae9bd5baa3fa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.590080 604 tablet_replica.cc:330] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: stopping tablet replica
I20260808 02:22:32.590180 604 raft_consensus.cc:2229] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:32.590543 604 raft_consensus.cc:2258] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:32.591009 604 ts_tablet_manager.cc:1905] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:32.591176 580 tablet_bootstrap.cc:492] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Bootstrap starting.
I20260808 02:22:32.591609 32662 tablet_service.cc:1467] Processing CreateTablet for tablet a330d280ac284e3fa4ae9bd5baa3fa80 (DEFAULT_TABLE table=test-table [id=b8ec4dfedd384e43908c8b7c4fb78ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:32.591638 580 tablet_bootstrap.cc:654] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.591718 32662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a330d280ac284e3fa4ae9bd5baa3fa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.592662 604 ts_tablet_manager.cc:1918] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:32.592726 604 log.cc:1196] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/45791ee52ade42c3964f64a5b870c5f6
I20260808 02:22:32.593000 604 ts_tablet_manager.cc:1939] T 45791ee52ade42c3964f64a5b870c5f6 P b7d435d3b4b84dd497f804240c6afcc6: Deleting consensus metadata
I20260808 02:22:32.593075 581 tablet_bootstrap.cc:492] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Bootstrap starting.
I20260808 02:22:32.593477 581 tablet_bootstrap.cc:654] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.594116 581 tablet_bootstrap.cc:492] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: No bootstrap required, opened a new log
I20260808 02:22:32.594101 32505 catalog_manager.cc:4841] TS b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739): tablet 45791ee52ade42c3964f64a5b870c5f6 (table test-table [id=49d2b26e11604ef289c26dfd3db97660]) successfully deleted
I20260808 02:22:32.594177 581 ts_tablet_manager.cc:1397] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:32.594323 581 raft_consensus.cc:348] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.594359 601 raft_consensus.cc:2258] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:32.594396 581 raft_consensus.cc:374] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.594419 581 raft_consensus.cc:729] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d435d3b4b84dd497f804240c6afcc6, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.594462 581 consensus_queue.cc:260] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.594551 581 ts_tablet_manager.cc:1428] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.594986 601 ts_tablet_manager.cc:1905] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:32.596062 603 ts_tablet_manager.cc:1918] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:32.596154 603 log.cc:1196] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/45791ee52ade42c3964f64a5b870c5f6
I20260808 02:22:32.596457 603 ts_tablet_manager.cc:1939] T 45791ee52ade42c3964f64a5b870c5f6 P 73d11b9ebab54dbe8486548d752d2c30: Deleting consensus metadata
I20260808 02:22:32.596587 601 ts_tablet_manager.cc:1918] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:32.596681 601 log.cc:1196] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/45791ee52ade42c3964f64a5b870c5f6
I20260808 02:22:32.596916 578 tablet_bootstrap.cc:492] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Bootstrap starting.
I20260808 02:22:32.596969 601 ts_tablet_manager.cc:1939] T 45791ee52ade42c3964f64a5b870c5f6 P ea1d5653f326471496ac8d315b1d1360: Deleting consensus metadata
I20260808 02:22:32.597141 580 tablet_bootstrap.cc:492] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: No bootstrap required, opened a new log
I20260808 02:22:32.597213 580 ts_tablet_manager.cc:1397] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260808 02:22:32.597327 578 tablet_bootstrap.cc:654] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.597364 32506 catalog_manager.cc:4841] TS ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233): tablet 45791ee52ade42c3964f64a5b870c5f6 (table test-table [id=49d2b26e11604ef289c26dfd3db97660]) successfully deleted
I20260808 02:22:32.597397 580 raft_consensus.cc:348] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.597479 580 raft_consensus.cc:374] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.597512 580 raft_consensus.cc:729] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1d5653f326471496ac8d315b1d1360, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.597563 580 consensus_queue.cc:260] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.597657 580 ts_tablet_manager.cc:1428] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.597693 32507 catalog_manager.cc:4841] TS 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2:40687): tablet 45791ee52ade42c3964f64a5b870c5f6 (table test-table [id=49d2b26e11604ef289c26dfd3db97660]) successfully deleted
I20260808 02:22:32.598193 578 tablet_bootstrap.cc:492] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: No bootstrap required, opened a new log
I20260808 02:22:32.598249 578 ts_tablet_manager.cc:1397] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:32.598410 578 raft_consensus.cc:348] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.598508 578 raft_consensus.cc:374] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.598531 578 raft_consensus.cc:729] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d11b9ebab54dbe8486548d752d2c30, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.598579 578 consensus_queue.cc:260] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.598696 578 ts_tablet_manager.cc:1428] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260808 02:22:32.688815 32732 tablet.cc:2369] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:22:32.753923 555 tablet.cc:2369] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:22:32.845450 408 tablet.cc:2369] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:32.904736 588 raft_consensus.cc:482] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:32.904886 588 raft_consensus.cc:504] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.905086 588 leader_election.cc:288] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739), ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233)
I20260808 02:22:32.905337 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a330d280ac284e3fa4ae9bd5baa3fa80" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1d5653f326471496ac8d315b1d1360" is_pre_election: true
I20260808 02:22:32.905373 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a330d280ac284e3fa4ae9bd5baa3fa80" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d435d3b4b84dd497f804240c6afcc6" is_pre_election: true
I20260808 02:22:32.905447 506 raft_consensus.cc:2454] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 0.
I20260808 02:22:32.905481 32683 raft_consensus.cc:2454] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 0.
I20260808 02:22:32.905639 32757 leader_election.cc:302] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d11b9ebab54dbe8486548d752d2c30, ea1d5653f326471496ac8d315b1d1360; no voters:
I20260808 02:22:32.905756 588 raft_consensus.cc:2781] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:32.905805 588 raft_consensus.cc:482] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:32.905831 588 raft_consensus.cc:3037] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.906626 588 raft_consensus.cc:504] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.906766 588 leader_election.cc:288] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 election: Requested vote from peers b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739), ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233)
I20260808 02:22:32.906942 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a330d280ac284e3fa4ae9bd5baa3fa80" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d435d3b4b84dd497f804240c6afcc6"
I20260808 02:22:32.906955 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a330d280ac284e3fa4ae9bd5baa3fa80" candidate_uuid: "73d11b9ebab54dbe8486548d752d2c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1d5653f326471496ac8d315b1d1360"
I20260808 02:22:32.907028 32683 raft_consensus.cc:3037] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.907040 506 raft_consensus.cc:3037] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:32.907671 506 raft_consensus.cc:2454] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 1.
I20260808 02:22:32.907722 32683 raft_consensus.cc:2454] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d11b9ebab54dbe8486548d752d2c30 in term 1.
I20260808 02:22:32.907930 32757 leader_election.cc:302] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d11b9ebab54dbe8486548d752d2c30, ea1d5653f326471496ac8d315b1d1360; no voters:
I20260808 02:22:32.908048 588 raft_consensus.cc:2781] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:32.908164 588 raft_consensus.cc:686] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 LEADER]: Becoming Leader. State: Replica: 73d11b9ebab54dbe8486548d752d2c30, State: Running, Role: LEADER
I20260808 02:22:32.908252 588 consensus_queue.cc:237] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.908919 32520 catalog_manager.cc:5495] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "73d11b9ebab54dbe8486548d752d2c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:22:32 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155751, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:32.975405 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.968886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.30.160.62:37631)
0808 02:22:32.968966 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:32.968968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:32.969087 (+ 119us) server_negotiation.cc:407] Connection header received
0808 02:22:32.969117 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:32.969118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:32.969128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:32.969190 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:32.969667 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.970158 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.970825 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.970998 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.971048 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.973140 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:32.973142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:32.973142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:32.973149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:32.974350 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.974685 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.974686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.974686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.974734 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:32.974924 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:32.974924 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:32.974925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:32.975020 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:32.975069 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:32.975205 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 02:22:32.975308 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":47}
I20260808 02:22:32.975395 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:32.968832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37631 (local address 127.0.0.1:45644)
0808 02:22:32.968994 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:32.969007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:32.969078 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:32.969206 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:32.969206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:32.969542 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:32.969590 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.969592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.970226 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.970227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.970742 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:32.970744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:32.971062 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:32.971063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:32.971104 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:32.971678 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:32.971683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:32.973063 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:32.974423 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.974430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.974432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.974621 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.974801 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:32.974802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:32.974802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:32.974870 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:32.975076 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:32.975082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:32.975122 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:22:32.975174 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 02:22:32.975261 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260808 02:22:32.977547 32520 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:45644:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:22:32.977772 32520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:32.979245 32662 tablet_service.cc:1467] Processing CreateTablet for tablet c1938ba60e11494d86757933ae45ed4f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e98e73079bb4b249b07aef411f7984a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:22:32.979238 338 tablet_service.cc:1467] Processing CreateTablet for tablet c1938ba60e11494d86757933ae45ed4f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e98e73079bb4b249b07aef411f7984a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:22:32.979291 485 tablet_service.cc:1467] Processing CreateTablet for tablet c1938ba60e11494d86757933ae45ed4f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e98e73079bb4b249b07aef411f7984a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:22:32.979423 485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1938ba60e11494d86757933ae45ed4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.979421 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1938ba60e11494d86757933ae45ed4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.979436 32662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1938ba60e11494d86757933ae45ed4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:32.980682 581 tablet_bootstrap.cc:492] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: Bootstrap starting.
I20260808 02:22:32.981022 578 tablet_bootstrap.cc:492] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: Bootstrap starting.
I20260808 02:22:32.981132 581 tablet_bootstrap.cc:654] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.981452 578 tablet_bootstrap.cc:654] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.981740 580 tablet_bootstrap.cc:492] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: Bootstrap starting.
I20260808 02:22:32.981899 581 tablet_bootstrap.cc:492] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: No bootstrap required, opened a new log
I20260808 02:22:32.981954 581 ts_tablet_manager.cc:1397] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:32.982043 578 tablet_bootstrap.cc:492] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: No bootstrap required, opened a new log
I20260808 02:22:32.982082 578 ts_tablet_manager.cc:1397] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:32.982151 580 tablet_bootstrap.cc:654] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:32.982120 581 raft_consensus.cc:348] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.982221 581 raft_consensus.cc:374] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.982190 578 raft_consensus.cc:348] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.982244 581 raft_consensus.cc:729] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d435d3b4b84dd497f804240c6afcc6, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.982256 578 raft_consensus.cc:374] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.982278 578 raft_consensus.cc:729] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d11b9ebab54dbe8486548d752d2c30, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.982293 581 consensus_queue.cc:260] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.982337 578 consensus_queue.cc:260] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.982400 581 ts_tablet_manager.cc:1428] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.982447 578 ts_tablet_manager.cc:1428] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.982479 585 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:32.982533 588 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:32.982589 588 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:32.982590 585 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:32.982865 580 tablet_bootstrap.cc:492] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: No bootstrap required, opened a new log
I20260808 02:22:32.982919 580 ts_tablet_manager.cc:1397] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:32.983084 580 raft_consensus.cc:348] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.983165 580 raft_consensus.cc:374] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:32.983214 580 raft_consensus.cc:729] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1d5653f326471496ac8d315b1d1360, State: Initialized, Role: FOLLOWER
I20260808 02:22:32.983255 580 consensus_queue.cc:260] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:32.983363 580 ts_tablet_manager.cc:1428] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:32.983454 586 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:32.983525 586 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:33.046624 585 raft_consensus.cc:482] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:33.046736 585 raft_consensus.cc:504] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:33.047070 585 leader_election.cc:288] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2:40687), ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233)
I20260808 02:22:33.051385 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.047346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:50447 (local address 127.30.160.3:41233)
0808 02:22:33.047402 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.047404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.047466 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:22:33.047531 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.047532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.047550 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.047613 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:33.047866 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.048664 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.050313 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.050885 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.050959 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.050967 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.051272 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.051313 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 02:22:33.051386 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.047421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:50073 (local address 127.30.160.2:40687)
0808 02:22:33.047472 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.047475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.047487 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:22:33.047540 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.047542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.047553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.047613 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:33.047865 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.048665 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.050320 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.050877 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.050959 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.050967 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.051249 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.051298 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:33.051414 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.047255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:41233 (local address 127.30.160.1:50447)
0808 02:22:33.047390 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:33.047403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:33.047463 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:33.047635 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:33.047636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:33.047639 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:33.047749 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.047752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.048758 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.048758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.050220 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.050222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.051096 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.051097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.051177 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.051181 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:33.051255 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:22:33.051287 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":98,"thread_start_us":76,"threads_started":1}
I20260808 02:22:33.051414 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.047255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:40687 (local address 127.30.160.1:50073)
0808 02:22:33.047391 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:33.047403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:33.047463 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:33.047635 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:33.047636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:33.047639 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:33.047759 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.047761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.048757 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.048758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.050219 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.050222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.051031 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.051033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.051143 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.051148 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:33.051219 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:22:33.051280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":79,"thread_start_us":51,"threads_started":1}
I20260808 02:22:33.051774 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1938ba60e11494d86757933ae45ed4f" candidate_uuid: "b7d435d3b4b84dd497f804240c6afcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1d5653f326471496ac8d315b1d1360" is_pre_election: true
I20260808 02:22:33.051877 506 raft_consensus.cc:2454] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7d435d3b4b84dd497f804240c6afcc6 in term 0.
I20260808 02:22:33.051853 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1938ba60e11494d86757933ae45ed4f" candidate_uuid: "b7d435d3b4b84dd497f804240c6afcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d11b9ebab54dbe8486548d752d2c30" is_pre_election: true
I20260808 02:22:33.052042 359 raft_consensus.cc:2454] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7d435d3b4b84dd497f804240c6afcc6 in term 0.
I20260808 02:22:33.052119 32614 leader_election.cc:302] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7d435d3b4b84dd497f804240c6afcc6, ea1d5653f326471496ac8d315b1d1360; no voters:
I20260808 02:22:33.052304 585 raft_consensus.cc:2781] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:33.052346 585 raft_consensus.cc:482] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:33.052361 585 raft_consensus.cc:3037] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:33.053200 585 raft_consensus.cc:504] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:33.053371 585 leader_election.cc:288] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [CANDIDATE]: Term 1 election: Requested vote from peers 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2:40687), ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233)
I20260808 02:22:33.053546 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1938ba60e11494d86757933ae45ed4f" candidate_uuid: "b7d435d3b4b84dd497f804240c6afcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1d5653f326471496ac8d315b1d1360"
I20260808 02:22:33.053624 506 raft_consensus.cc:3037] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:33.054235 506 raft_consensus.cc:2454] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d435d3b4b84dd497f804240c6afcc6 in term 1.
I20260808 02:22:33.054375 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1938ba60e11494d86757933ae45ed4f" candidate_uuid: "b7d435d3b4b84dd497f804240c6afcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d11b9ebab54dbe8486548d752d2c30"
I20260808 02:22:33.054425 32614 leader_election.cc:302] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7d435d3b4b84dd497f804240c6afcc6, ea1d5653f326471496ac8d315b1d1360; no voters:
I20260808 02:22:33.054461 359 raft_consensus.cc:3037] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:33.054527 585 raft_consensus.cc:2781] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:33.054693 585 raft_consensus.cc:686] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [term 1 LEADER]: Becoming Leader. State: Replica: b7d435d3b4b84dd497f804240c6afcc6, State: Running, Role: LEADER
I20260808 02:22:33.054776 585 consensus_queue.cc:237] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } }
I20260808 02:22:33.055147 621 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: TxnStatusTablet state changed. Reason: New leader b7d435d3b4b84dd497f804240c6afcc6. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.055284 621 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:22:33.055367 359 raft_consensus.cc:2454] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d435d3b4b84dd497f804240c6afcc6 in term 1.
I20260808 02:22:33.055550 623 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:22:33.055531 32521 catalog_manager.cc:5495] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1). New cstate: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:33.055653 623 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:22:33.076752 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.074041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.30.160.1:42739)
0808 02:22:33.074175 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.074177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.074186 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:33.074225 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.074226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.074245 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.074350 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:33.074506 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.075029 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.075788 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.076098 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.076176 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.076369 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:33.076492 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:33.076577 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.076636 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.076656 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260808 02:22:33.076756 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.073979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42739 (local address 127.0.0.1:38480)
0808 02:22:33.074056 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:33.074068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:33.074146 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:33.074358 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:33.074359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:33.074360 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:33.074417 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.074419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.075118 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.075119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.075700 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.075702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.076175 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.076176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.076246 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.076268 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:33.076562 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:33.076574 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:22:33.076630 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:22:33.076683 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 02:22:33.078136 506 raft_consensus.cc:1261] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Refusing update from remote peer b7d435d3b4b84dd497f804240c6afcc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:33.078161 359 raft_consensus.cc:1261] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Refusing update from remote peer b7d435d3b4b84dd497f804240c6afcc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:33.078413 621 consensus_queue.cc:1035] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:33.078547 585 consensus_queue.cc:1035] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:33.079167 588 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: TxnStatusTablet state changed. Reason: New leader b7d435d3b4b84dd497f804240c6afcc6. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.079258 588 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:33.079497 586 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: TxnStatusTablet state changed. Reason: New leader b7d435d3b4b84dd497f804240c6afcc6. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.079598 586 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:33.079810 621 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.079944 621 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:22:33.080175 585 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.080253 585 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P b7d435d3b4b84dd497f804240c6afcc6: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:22:33.080289 588 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.080369 588 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P 73d11b9ebab54dbe8486548d752d2c30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:33.080826 586 tablet_replica.cc:439] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7d435d3b4b84dd497f804240c6afcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 } } peers { permanent_uuid: "73d11b9ebab54dbe8486548d752d2c30" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 40687 } } peers { permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 } } }
I20260808 02:22:33.080899 586 tablet_replica.cc:442] T c1938ba60e11494d86757933ae45ed4f P ea1d5653f326471496ac8d315b1d1360: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:33.086105 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.083649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:48507 (local address 127.30.160.1:42739)
0808 02:22:33.083705 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.083707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.083768 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:22:33.083851 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.083852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.083862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.083932 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:33.084053 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.084597 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.085373 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.085615 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.085695 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.085879 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:33.085909 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:33.085963 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.086024 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.086043 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:33.086146 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.083587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42739 (local address 127.30.160.2:48507)
0808 02:22:33.083712 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:33.083724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:33.083786 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:33.083936 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:33.083937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:33.083938 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:33.083987 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.083989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.084684 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.084685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.085272 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.085274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.085695 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.085696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.085754 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.085772 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:33.085967 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:33.085968 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:22:33.086019 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 02:22:33.086066 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260808 02:22:33.088651 32683 raft_consensus.cc:1261] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Refusing update from remote peer 73d11b9ebab54dbe8486548d752d2c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:33.088694 506 raft_consensus.cc:1261] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Refusing update from remote peer 73d11b9ebab54dbe8486548d752d2c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:33.088884 584 consensus_queue.cc:1035] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d435d3b4b84dd497f804240c6afcc6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:33.088972 588 consensus_queue.cc:1035] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1d5653f326471496ac8d315b1d1360" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:33.096422 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.093911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:40687 (local address 127.30.160.1:56933)
0808 02:22:33.094018 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:33.094030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:33.094083 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:33.094201 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:33.094202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:33.094203 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:33.094252 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.094253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.094854 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.094855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.095456 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:33.095457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.095867 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.095868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:33.095954 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.095974 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:33.096189 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:33.096191 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:22:33.096255 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 02:22:33.096307 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":52,"threads_started":1}
I20260808 02:22:33.096686 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.094059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:56933 (local address 127.30.160.2:40687)
0808 02:22:33.094097 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.094099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.094108 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:33.094131 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.094132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.094142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.094196 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:33.094300 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.094755 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.095544 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.095792 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.095867 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.096076 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:33.096112 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:33.096143 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.096599 (+ 456us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.096625 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:33.105093 32520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:33.105189 32520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:33.105858 32520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92d28138f7648af8e6d1e4e7a5e639b: Sending DeleteTablet for 3 replicas of tablet a330d280ac284e3fa4ae9bd5baa3fa80
I20260808 02:22:33.106086 338 tablet_service.cc:1514] Processing DeleteTablet for tablet a330d280ac284e3fa4ae9bd5baa3fa80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:33 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:44478
I20260808 02:22:33.106117 485 tablet_service.cc:1514] Processing DeleteTablet for tablet a330d280ac284e3fa4ae9bd5baa3fa80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:33 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:52686
I20260808 02:22:33.106240 32662 tablet_service.cc:1514] Processing DeleteTablet for tablet a330d280ac284e3fa4ae9bd5baa3fa80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:33 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:38476
I20260808 02:22:33.106258 641 tablet_replica.cc:330] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: stopping tablet replica
I20260808 02:22:33.106325 641 raft_consensus.cc:2229] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:33.106370 641 raft_consensus.cc:2258] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:33.106459 640 tablet_replica.cc:330] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: stopping tablet replica
I20260808 02:22:33.106514 640 raft_consensus.cc:2229] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:33.106575 640 raft_consensus.cc:2258] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:33.106841 640 ts_tablet_manager.cc:1905] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:33.107108 641 ts_tablet_manager.cc:1905] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:33.108702 640 ts_tablet_manager.cc:1918] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:33.108774 640 log.cc:1196] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/a330d280ac284e3fa4ae9bd5baa3fa80
I20260808 02:22:33.108798 641 ts_tablet_manager.cc:1918] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:33.108856 641 log.cc:1196] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/a330d280ac284e3fa4ae9bd5baa3fa80
I20260808 02:22:33.109026 640 ts_tablet_manager.cc:1939] T a330d280ac284e3fa4ae9bd5baa3fa80 P 73d11b9ebab54dbe8486548d752d2c30: Deleting consensus metadata
I20260808 02:22:33.109098 641 ts_tablet_manager.cc:1939] T a330d280ac284e3fa4ae9bd5baa3fa80 P ea1d5653f326471496ac8d315b1d1360: Deleting consensus metadata
I20260808 02:22:33.109378 32507 catalog_manager.cc:4841] TS 73d11b9ebab54dbe8486548d752d2c30 (127.30.160.2:40687): tablet a330d280ac284e3fa4ae9bd5baa3fa80 (table test-table [id=b8ec4dfedd384e43908c8b7c4fb78ee8]) successfully deleted
I20260808 02:22:33.109408 32506 catalog_manager.cc:4841] TS ea1d5653f326471496ac8d315b1d1360 (127.30.160.3:41233): tablet a330d280ac284e3fa4ae9bd5baa3fa80 (table test-table [id=b8ec4dfedd384e43908c8b7c4fb78ee8]) successfully deleted
I20260808 02:22:33.109583 642 tablet_replica.cc:330] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: stopping tablet replica
I20260808 02:22:33.109633 642 raft_consensus.cc:2229] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:33.109666 642 raft_consensus.cc:2258] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:33.109886 642 ts_tablet_manager.cc:1905] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:33.111918 642 ts_tablet_manager.cc:1918] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:33.111984 642 log.cc:1196] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/a330d280ac284e3fa4ae9bd5baa3fa80
I20260808 02:22:33.112206 642 ts_tablet_manager.cc:1939] T a330d280ac284e3fa4ae9bd5baa3fa80 P b7d435d3b4b84dd497f804240c6afcc6: Deleting consensus metadata
I20260808 02:22:33.112514 32505 catalog_manager.cc:4841] TS b7d435d3b4b84dd497f804240c6afcc6 (127.30.160.1:42739): tablet a330d280ac284e3fa4ae9bd5baa3fa80 (table test-table [id=b8ec4dfedd384e43908c8b7c4fb78ee8]) successfully deleted
Aug 08 02:22:33 dist-test-slave-zkpd krb5kdc[32476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:33.116287 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.107104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.30.160.1:42739)
0808 02:22:33.107146 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.107148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.108057 (+ 909us) server_negotiation.cc:407] Connection header received
0808 02:22:33.108180 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.108181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.108191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.108219 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:33.109299 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.109833 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.110614 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.110823 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.110900 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.113469 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:33.113479 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:33.113479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:33.113487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:33.115030 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:33.115410 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:33.115414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:33.115414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:33.115478 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:33.115692 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:33.115693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:33.115694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:33.115837 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:33.116053 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.116091 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.116167 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 02:22:33.116508 32725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38496
I20260808 02:22:33.123273 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:33.117455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.30.160.62:37631)
0808 02:22:33.117506 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:33.117508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:33.117516 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:33.117588 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:33.117589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:33.117599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:33.117653 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:33.118126 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.118625 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.119276 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:33.119483 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:33.119567 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:33.120709 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:33.120710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:33.120711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:33.120719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:33.122102 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:33.122456 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:33.122457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:33.122457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:33.122526 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:33.122730 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:33.122731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:33.122731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:33.122851 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:33.122955 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:33.123049 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:22:33.123128 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260808 02:22:33.123495 32520 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658
I20260808 02:22:33.124015 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 32597
I20260808 02:22:33.137373 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 32740
I20260808 02:22:33.144280 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 416
I20260808 02:22:33.150158 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 32489
2026-08-08T02:22:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2469 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:33 dist-test-slave-zkpd krb5kdc[649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:33 dist-test-slave-zkpd krb5kdc[649](info): set up 2 sockets
Aug 08 02:22:33 dist-test-slave-zkpd krb5kdc[649](info): commencing operation
krb5kdc: starting...
W20260808 02:22:34.227629 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:34.353039 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34919
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:34.434939 662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:34.435180 662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:34.435218 662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:34.436424 662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:34.436486 662 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:22:34.436509 662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:34.436543 662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:34.436586 662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:34.436647 662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:34.438179 662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:34919
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:34.438467 662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:34.438694 662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:34.443531 667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:34.443583 668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:34.443678 662 server_base.cc:1034] running on GCE node
W20260808 02:22:34.443861 670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:34.444156 662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:34.444469 662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:34.445644 662 hybrid_clock.cc:648] HybridClock initialized: now 1786155754445620 us; error 40 us; skew 500 ppm
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:34.448515 662 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:34.448945 662 webserver.cc:460] Webserver started at http://127.30.160.62:43781/ using document root <none> and password file <none>
I20260808 02:22:34.449194 662 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:34.449265 662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:34.449419 662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:34.450465 662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "4c1d3db6e96a462ea43735b00441ff9d"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "a11b9bcdeb016ab3b52566aeb3f9293d"
server_key_iv: "a84bb454268ae64bd17dabfbbf132f77"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.450836 662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "4c1d3db6e96a462ea43735b00441ff9d"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "a11b9bcdeb016ab3b52566aeb3f9293d"
server_key_iv: "a84bb454268ae64bd17dabfbbf132f77"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.452191 662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:34.452905 677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.453078 662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:34.453184 662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "4c1d3db6e96a462ea43735b00441ff9d"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "a11b9bcdeb016ab3b52566aeb3f9293d"
server_key_iv: "a84bb454268ae64bd17dabfbbf132f77"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.453260 662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:34.478514 662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:34.481050 662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:34.481226 662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:34.485945 662 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:34919
I20260808 02:22:34.485996 740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:34919 every 8 connection(s)
I20260808 02:22:34.486322 662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:34.486963 741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:34.487978 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 662
I20260808 02:22:34.488063 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:34.488233 31360 external_mini_cluster.cc:1428] Setting key 8b31b1e7c12b40999f0f4c8499d30317
I20260808 02:22:34.489663 741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Bootstrap starting.
I20260808 02:22:34.490270 741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:34.490569 741 log.cc:824] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:34.491258 741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: No bootstrap required, opened a new log
I20260808 02:22:34.493717 741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d3db6e96a462ea43735b00441ff9d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34919 } }
I20260808 02:22:34.493927 741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:34.493973 741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1d3db6e96a462ea43735b00441ff9d, State: Initialized, Role: FOLLOWER
I20260808 02:22:34.494122 741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d3db6e96a462ea43735b00441ff9d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34919 } }
I20260808 02:22:34.494220 741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:34.494257 741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:34.494305 741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:34.494941 741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d3db6e96a462ea43735b00441ff9d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34919 } }
I20260808 02:22:34.495033 741 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1d3db6e96a462ea43735b00441ff9d; no voters:
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:34.495222 741 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:34.495266 746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:34.495406 746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [term 1 LEADER]: Becoming Leader. State: Replica: 4c1d3db6e96a462ea43735b00441ff9d, State: Running, Role: LEADER
I20260808 02:22:34.495508 741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:34.495575 746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d3db6e96a462ea43735b00441ff9d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34919 } }
I20260808 02:22:34.496114 747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1d3db6e96a462ea43735b00441ff9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d3db6e96a462ea43735b00441ff9d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34919 } } }
I20260808 02:22:34.496212 747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:34.496449 748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1d3db6e96a462ea43735b00441ff9d. Latest consensus state: current_term: 1 leader_uuid: "4c1d3db6e96a462ea43735b00441ff9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d3db6e96a462ea43735b00441ff9d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34919 } } }
I20260808 02:22:34.496584 748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:34.496760 755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:34.497265 755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:34.499343 755 catalog_manager.cc:1347] Generated new cluster ID: 40a3e79ed46943efa03da76b13372ec5
I20260808 02:22:34.499395 755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:34.499723 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.488763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.30.160.62:34919)
0808 02:22:34.488954 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:34.488970 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:34.488995 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:22:34.489480 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:34.489486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:34.489531 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:34.489687 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:34.490332 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.491319 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.492502 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.492923 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.493002 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.495724 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:34.495728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:34.495729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:34.495741 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:34.497814 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.498240 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.498243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.498244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.498385 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.498636 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.498637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.498638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.498886 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:34.499005 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:34.499388 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 02:22:34.499530 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260808 02:22:34.509907 755 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:34.510408 755 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:34.523072 755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Generated new TSK 0
I20260808 02:22:34.523308 755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:34.551234 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34919
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:34.634539 767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:34.634737 767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:34.634763 767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:34.635978 767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:34.636016 767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:34.636037 767 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:22:34.636078 767 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:34.636111 767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:34.637688 767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:34.637936 767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:34.638170 767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:34.643280 772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:34.643288 773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:34.643301 775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:34.643463 767 server_base.cc:1034] running on GCE node
I20260808 02:22:34.643743 767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:34.644022 767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:34.645198 767 hybrid_clock.cc:648] HybridClock initialized: now 1786155754645183 us; error 34 us; skew 500 ppm
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:34.647670 767 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:34.648115 767 webserver.cc:460] Webserver started at http://127.30.160.1:33129/ using document root <none> and password file <none>
I20260808 02:22:34.648309 767 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:34.648350 767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:34.648439 767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:34.649380 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "18a884945bbe4cbaa058a1d3c5e27f3c"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "29186aa289ce2e95da45b5072e09e3c9"
server_key_iv: "18c0083909595100db8c38f1ca2f4b8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.649670 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "18a884945bbe4cbaa058a1d3c5e27f3c"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "29186aa289ce2e95da45b5072e09e3c9"
server_key_iv: "18c0083909595100db8c38f1ca2f4b8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.650941 767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:34.651615 782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.651804 767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.651893 767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "18a884945bbe4cbaa058a1d3c5e27f3c"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "29186aa289ce2e95da45b5072e09e3c9"
server_key_iv: "18c0083909595100db8c38f1ca2f4b8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.651993 767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:34.683414 767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:34.685865 767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:34.686035 767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:34.686640 767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:34.686683 767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.686708 767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:34.686721 767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.693634 767 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:38087
I20260808 02:22:34.693734 901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:38087 every 8 connection(s)
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:34.694111 767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:34.696835 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 767
I20260808 02:22:34.696961 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:34.697177 31360 external_mini_cluster.cc:1428] Setting key 03324088a3e404bff06f9f2d0423c9e3
I20260808 02:22:34.697206 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.687152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:33393 (local address 127.30.160.62:34919)
0808 02:22:34.687231 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:34.687234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:34.687407 (+ 173us) server_negotiation.cc:407] Connection header received
0808 02:22:34.687932 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:34.687933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:34.687946 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:34.688029 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:34.688800 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.689656 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.691254 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.691636 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.691738 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.694599 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:34.694601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:34.694601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:34.694609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:34.695930 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.696345 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.696346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.696346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.696399 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.696571 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.696571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.696572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.696668 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:34.696714 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:34.697066 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 02:22:34.697141 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:22:34.698671 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.687083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.30.160.1:33393)
0808 02:22:34.687260 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:34.687300 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:34.687810 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:34.688101 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:34.688104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:34.688386 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:34.688687 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.688694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.689776 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.689777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.691147 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.691150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.691764 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.691765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.691939 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.692493 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:34.692503 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:34.694452 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:34.696011 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.696018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.696020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.696278 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.696444 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.696444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.696444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.696534 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.696713 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:34.696720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:34.696886 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 02:22:34.697036 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 02:22:34.697224 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260808 02:22:34.699134 902 heartbeater.cc:347] Connected to a master server at 127.30.160.62:34919
I20260808 02:22:34.699252 902 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:34.699476 902 heartbeater.cc:511] Master 127.30.160.62:34919 requested a full tablet report, sending...
I20260808 02:22:34.700134 693 ts_manager.cc:194] Registered new tserver with Master: 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087)
I20260808 02:22:34.700975 693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:33393
I20260808 02:22:34.706204 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.700409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:36683 (local address 127.30.160.62:34919)
0808 02:22:34.700443 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:34.700444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:34.700453 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:34.700507 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:34.700508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:34.700517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:34.700575 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:34.700983 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.701620 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.702392 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.702600 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.702686 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.703625 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:34.703626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:34.703627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:34.703634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:34.705076 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.705469 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.705470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.705470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.705518 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.705766 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.705767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.705767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.705869 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:34.705919 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:34.706092 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:22:34.706149 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 02:22:34.706321 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.700231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.30.160.1:36683)
0808 02:22:34.700386 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:34.700399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:34.700454 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:34.700598 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:34.700599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:34.700827 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:34.700896 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.700898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.701758 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.701759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.702303 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.702305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.702688 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.702689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.702777 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.703254 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:34.703259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:34.703530 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:34.705161 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.705162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.705162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.705396 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.705582 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.705583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.705583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.705659 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.705931 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:34.705932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:34.705990 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:22:34.706076 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:22:34.706185 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:34.726995 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34919
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:34.809671 911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:34.809908 911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:34.809948 911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:34.811094 911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:34.811165 911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:34.811200 911 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:22:34.811254 911 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:34.811291 911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:34.812788 911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:34.813066 911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:34.813310 911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:34.818282 919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:34.818326 917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:34.818280 916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:34.818535 911 server_base.cc:1034] running on GCE node
I20260808 02:22:34.818750 911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:34.818996 911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:34.820137 911 hybrid_clock.cc:648] HybridClock initialized: now 1786155754820124 us; error 27 us; skew 500 ppm
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:34.826793 911 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:34.827262 911 webserver.cc:460] Webserver started at http://127.30.160.2:46317/ using document root <none> and password file <none>
I20260808 02:22:34.827508 911 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:34.827579 911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:34.827726 911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:34.828866 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "375ce3ceefd24c278478525e12b8fdb9"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "555105eb10490d5ff88af984091a42a0"
server_key_iv: "6c76adaa2be58f3d10d3f68d6183d9d4"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.829246 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "375ce3ceefd24c278478525e12b8fdb9"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "555105eb10490d5ff88af984091a42a0"
server_key_iv: "6c76adaa2be58f3d10d3f68d6183d9d4"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.830587 911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:34.831331 926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.831543 911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:34.831664 911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "375ce3ceefd24c278478525e12b8fdb9"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "555105eb10490d5ff88af984091a42a0"
server_key_iv: "6c76adaa2be58f3d10d3f68d6183d9d4"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.831744 911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:34.852224 911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:34.854678 911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:34.854844 911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:34.855463 911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:34.855520 911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.855556 911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:34.855616 911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:34.862948 911 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:45247
I20260808 02:22:34.863027 1044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:45247 every 8 connection(s)
I20260808 02:22:34.863369 911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:34.865803 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.855941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:37691 (local address 127.30.160.62:34919)
0808 02:22:34.856008 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:34.856011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:34.856233 (+ 222us) server_negotiation.cc:407] Connection header received
0808 02:22:34.856846 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:34.856847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:34.856860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:34.856943 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:34.857750 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.858594 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.860175 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.860483 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.860571 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.862909 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:34.862911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:34.862911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:34.862919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:34.864404 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.864831 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.864832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.864833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.864891 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.865159 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.865160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.865160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.865283 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:34.865359 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:34.865630 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:22:34.865712 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:22:34.866003 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.855913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.30.160.2:37691)
0808 02:22:34.856096 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:34.856138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:34.856745 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:34.857032 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:34.857044 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:34.857326 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:34.857641 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.857648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.858693 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.858694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.860075 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.860078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.860595 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.860596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.860732 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.861217 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:34.861224 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:34.862795 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:34.864506 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.864512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.864515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.864733 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.864968 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.864969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.864969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.865085 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.865351 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:34.865358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:34.865496 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 02:22:34.865605 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:22:34.865762 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260808 02:22:34.866511 1045 heartbeater.cc:347] Connected to a master server at 127.30.160.62:34919
I20260808 02:22:34.866638 1045 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:34.866818 1045 heartbeater.cc:511] Master 127.30.160.62:34919 requested a full tablet report, sending...
I20260808 02:22:34.867204 694 ts_manager.cc:194] Registered new tserver with Master: 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247)
I20260808 02:22:34.867623 694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:37691
I20260808 02:22:34.872385 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 911
I20260808 02:22:34.872484 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:34.872666 31360 external_mini_cluster.cc:1428] Setting key 7f7b2fc13a632775d2a0d3ae2330688a
I20260808 02:22:34.872799 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.867644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:47825 (local address 127.30.160.62:34919)
0808 02:22:34.867686 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:34.867688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:34.867774 (+ 86us) server_negotiation.cc:407] Connection header received
0808 02:22:34.867876 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:34.867877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:34.867887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:34.867962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:34.868288 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.868751 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.869452 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.869654 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.869738 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.870517 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:34.870518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:34.870519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:34.870525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:34.871597 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.871979 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.871980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.871981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.872035 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:34.872267 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:34.872268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:34.872268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:34.872421 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:34.872495 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:34.872637 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 02:22:34.872711 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:34.872816 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:34.867582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.30.160.2:47825)
0808 02:22:34.867699 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:34.867709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:34.867786 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:34.867982 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:34.867983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:34.868153 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:34.868203 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.868205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.868854 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.868855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.869356 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:34.869358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:34.869740 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:34.869741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:34.869821 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:34.870169 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:34.870173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:34.870427 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:34.871681 (+ 1254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.871682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.871683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.871860 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.872119 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:34.872119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:34.872120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:34.872198 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:34.872498 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:34.872500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:34.872547 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:22:34.872612 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:22:34.872712 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:34.898663 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34919
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:34.981102 1054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:34.981323 1054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:34.981365 1054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:34.982489 1054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:34.982553 1054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:34.982589 1054 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:22:34.982618 1054 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:34.982651 1054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:34.984115 1054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:34.984376 1054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:34.984606 1054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:34.989638 1062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:34.989771 1054 server_base.cc:1034] running on GCE node
W20260808 02:22:34.989617 1059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:34.989881 1060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:34.990106 1054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:34.990386 1054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:34.991566 1054 hybrid_clock.cc:648] HybridClock initialized: now 1786155754991545 us; error 36 us; skew 500 ppm
Aug 08 02:22:34 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:34.994081 1054 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:34.994500 1054 webserver.cc:460] Webserver started at http://127.30.160.3:38379/ using document root <none> and password file <none>
I20260808 02:22:34.994740 1054 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:34.994809 1054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:34.994964 1054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:34.996217 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "e45a7827306e497b827b855eba7a5227"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "a1edbde86673de35c647c90c19fcccd1"
server_key_iv: "17d38f55b35b6f4c76d38dab3e5b2644"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.996644 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "e45a7827306e497b827b855eba7a5227"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "a1edbde86673de35c647c90c19fcccd1"
server_key_iv: "17d38f55b35b6f4c76d38dab3e5b2644"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.998070 1054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:34.998837 1069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:34.999019 1054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:34.999119 1054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "e45a7827306e497b827b855eba7a5227"
format_stamp: "Formatted at 2026-08-08 02:22:34 on dist-test-slave-zkpd"
server_key: "a1edbde86673de35c647c90c19fcccd1"
server_key_iv: "17d38f55b35b6f4c76d38dab3e5b2644"
server_key_version: "encryptionkey@0"
I20260808 02:22:34.999205 1054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:35.021401 1054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:35.024011 1054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:35.024175 1054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:35.024706 1054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:35.024773 1054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:35.024819 1054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:35.024849 1054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:22:35 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:35.032038 1054 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:38905
I20260808 02:22:35.032122 1186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:38905 every 8 connection(s)
I20260808 02:22:35.032454 1054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:35.033560 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 1054
I20260808 02:22:35.033660 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:35.033890 31360 external_mini_cluster.cc:1428] Setting key 8bc797c24c59f41fec6de32633d6e6fb
I20260808 02:22:35.035574 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.025224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:38541 (local address 127.30.160.62:34919)
0808 02:22:35.025269 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.025271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.025506 (+ 235us) server_negotiation.cc:407] Connection header received
0808 02:22:35.026061 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.026063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.026076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.026134 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:35.026993 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.027784 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.029536 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.029782 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.029878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.032193 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:35.032195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:35.032196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:35.032205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:35.034041 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.034582 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.034583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.034583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.034651 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.034934 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.034935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.034935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.035055 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:35.035111 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.035377 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.035448 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:35.035856 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.025221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.30.160.3:38541)
0808 02:22:35.025377 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.025413 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.025950 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.026218 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.026222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.026611 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:35.026903 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.026911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.027949 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.027951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.029418 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.029421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.029893 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.029894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.030036 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.030577 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:35.030586 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:35.032078 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:35.034122 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:35.034130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:35.034132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:35.034512 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:35.034721 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:35.034722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:35.034722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:35.034861 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:35.035131 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:35.035136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:35.035252 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.035343 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.035464 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260808 02:22:35.036365 1187 heartbeater.cc:347] Connected to a master server at 127.30.160.62:34919
I20260808 02:22:35.036453 1187 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:35.036639 1187 heartbeater.cc:511] Master 127.30.160.62:34919 requested a full tablet report, sending...
I20260808 02:22:35.037027 693 ts_manager.cc:194] Registered new tserver with Master: e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905)
I20260808 02:22:35.037475 693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:38541
I20260808 02:22:35.042869 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.037493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:50601 (local address 127.30.160.62:34919)
0808 02:22:35.037546 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.037548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.037588 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:22:35.037658 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.037659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.037670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.037719 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:35.038084 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.038543 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.039242 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.039438 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.039524 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.040479 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:35.040480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:35.040481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:35.040488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:35.041714 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.042069 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.042070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.042070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.042130 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.042347 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.042348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.042348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.042459 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:35.042536 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.042694 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.042769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:22:35.042878 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.037372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.30.160.3:50601)
0808 02:22:35.037526 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.037538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.037604 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.037721 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.037722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.037937 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:35.038003 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.038005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.038638 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.038639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.039157 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.039158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.039527 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.039528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.039607 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.040098 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:35.040102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:35.040390 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:35.041811 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:35.041812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:35.041813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:35.041986 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:35.042197 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:35.042198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:35.042198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:35.042257 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:35.042529 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:35.042529 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:35.042578 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.042668 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.042771 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260808 02:22:35.045651 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:22:35 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:22:35 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:35.070447 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.062179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.30.160.62:34919)
0808 02:22:35.062245 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.062247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.062258 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:35.062309 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.062310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.062326 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.062390 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:35.062949 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.063435 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.064504 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.064860 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.064926 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.067586 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:35.067587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:35.067588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:35.067595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:35.069266 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.069623 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.069624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.069625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.069685 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.069920 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.069921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.069921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.070039 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:35.070145 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.070257 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.070340 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:22:35.071969 693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:35.073518 693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:35.080428 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.076417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.30.160.1:38087)
0808 02:22:35.076584 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.076589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.076600 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:35.076644 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.076647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.076664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.076772 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.076963 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.078136 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.079422 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.079992 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.080089 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.080152 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.080295 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.080333 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260808 02:22:35.080416 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.076062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:38087 (local address 127.0.0.1:49510)
0808 02:22:35.076285 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.076294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.076353 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.076785 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.076788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.076790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.076859 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.076861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.078233 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.078234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.079357 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.079359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.080091 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.080092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.080168 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.080181 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.080276 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.080331 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":31,"thread_start_us":42,"threads_started":1}
I20260808 02:22:35.080529 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.076062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:45247 (local address 127.0.0.1:38574)
0808 02:22:35.076241 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.076284 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.076353 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.076900 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.076900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.076903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.076945 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.076946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.078241 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.078242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.079201 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.079204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.080234 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.080235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.080298 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.080302 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.080362 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.080454 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":84,"thread_start_us":103,"threads_started":1}
I20260808 02:22:35.080542 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.076189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.30.160.2:45247)
0808 02:22:35.076634 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.076638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.076647 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:35.076684 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.076688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.076702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.076776 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.077032 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.078146 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.079290 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.080064 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.080137 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.080196 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.080407 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.080450 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":112,"threads_started":1}
I20260808 02:22:35.080681 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.076078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:38905 (local address 127.0.0.1:58190)
0808 02:22:35.076456 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.076464 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.076504 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.077086 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.077086 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.077088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.077139 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.077140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.077993 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.077994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.079289 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.079292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.080252 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.080252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.080308 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.080312 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.080362 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.080416 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260808 02:22:35.080899 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.076319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.30.160.3:38905)
0808 02:22:35.076934 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.076939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.076949 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:35.076990 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.076992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.077008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.077089 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.077203 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.077894 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.079366 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.080120 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.080175 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.080224 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.080736 (+ 512us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.080783 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":73,"threads_started":1}
I20260808 02:22:35.081308 976 tablet_service.cc:1467] Processing CreateTablet for tablet a7cb7c3c883045db91e2642a0ba3fbc7 (DEFAULT_TABLE table=test-table [id=9a707be4f5bb4c8c8da7da7b452b37d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:35.081554 1120 tablet_service.cc:1467] Processing CreateTablet for tablet a7cb7c3c883045db91e2642a0ba3fbc7 (DEFAULT_TABLE table=test-table [id=9a707be4f5bb4c8c8da7da7b452b37d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:35.081696 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cb7c3c883045db91e2642a0ba3fbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.081671 833 tablet_service.cc:1467] Processing CreateTablet for tablet a7cb7c3c883045db91e2642a0ba3fbc7 (DEFAULT_TABLE table=test-table [id=9a707be4f5bb4c8c8da7da7b452b37d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:35.081940 1120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cb7c3c883045db91e2642a0ba3fbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.082034 833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cb7c3c883045db91e2642a0ba3fbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.084622 1210 tablet_bootstrap.cc:492] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Bootstrap starting.
I20260808 02:22:35.085597 1211 tablet_bootstrap.cc:492] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Bootstrap starting.
I20260808 02:22:35.085697 1210 tablet_bootstrap.cc:654] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.086081 1210 log.cc:824] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:35.086274 1211 tablet_bootstrap.cc:654] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.086632 1211 log.cc:824] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:35.086959 1210 tablet_bootstrap.cc:492] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: No bootstrap required, opened a new log
I20260808 02:22:35.086980 1212 tablet_bootstrap.cc:492] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Bootstrap starting.
I20260808 02:22:35.087036 1210 ts_tablet_manager.cc:1397] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:35.087316 1211 tablet_bootstrap.cc:492] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: No bootstrap required, opened a new log
I20260808 02:22:35.087399 1211 ts_tablet_manager.cc:1397] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:35.087594 1212 tablet_bootstrap.cc:654] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.087980 1212 log.cc:824] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:35.088804 1212 tablet_bootstrap.cc:492] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: No bootstrap required, opened a new log
I20260808 02:22:35.088891 1212 ts_tablet_manager.cc:1397] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:22:35.089269 1211 raft_consensus.cc:348] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.089422 1211 raft_consensus.cc:374] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.089465 1211 raft_consensus.cc:729] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.089583 1211 consensus_queue.cc:260] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.089557 1210 raft_consensus.cc:348] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.089717 1210 raft_consensus.cc:374] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.089754 1210 raft_consensus.cc:729] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45a7827306e497b827b855eba7a5227, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.089867 1211 ts_tablet_manager.cc:1428] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 02:22:35.089869 1210 consensus_queue.cc:260] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.089991 902 heartbeater.cc:503] Master 127.30.160.62:34919 was elected leader, sending a full tablet report...
I20260808 02:22:35.090139 1210 ts_tablet_manager.cc:1428] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:22:35.090260 1187 heartbeater.cc:503] Master 127.30.160.62:34919 was elected leader, sending a full tablet report...
I20260808 02:22:35.091012 1212 raft_consensus.cc:348] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.091169 1212 raft_consensus.cc:374] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.091195 1212 raft_consensus.cc:729] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375ce3ceefd24c278478525e12b8fdb9, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.091322 1212 consensus_queue.cc:260] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.091590 1212 ts_tablet_manager.cc:1428] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:22:35.091668 1045 heartbeater.cc:503] Master 127.30.160.62:34919 was elected leader, sending a full tablet report...
I20260808 02:22:35.096146 1217 raft_consensus.cc:482] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:35.096241 1217 raft_consensus.cc:504] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.096541 1217 leader_election.cc:288] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247), 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087)
I20260808 02:22:35.099627 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.096673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:45247 (local address 127.30.160.3:41707)
0808 02:22:35.096730 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.096742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.096786 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.096935 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.096936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.096937 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.097002 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.097004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.097817 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.097819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.098722 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.098725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.099345 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.099346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.099420 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.099424 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.099487 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.099526 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 02:22:35.099649 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.096803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:41707 (local address 127.30.160.2:45247)
0808 02:22:35.096842 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.096843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.096851 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:35.096874 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.096874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.096883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.096933 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.097084 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.097740 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.098802 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.099273 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.099334 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.099341 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.099499 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.099596 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:35.099964 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.096780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:58645 (local address 127.30.160.1:38087)
0808 02:22:35.096820 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.096822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.097070 (+ 248us) server_negotiation.cc:407] Connection header received
0808 02:22:35.097128 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.097129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.097157 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.097217 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.097455 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.098004 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.099069 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.099517 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.099580 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.099587 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.099848 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.099884 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:22:35.099980 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.096700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:38087 (local address 127.30.160.3:58645)
0808 02:22:35.097019 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.097029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.097080 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.097296 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.097297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.097299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.097391 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.097393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.098085 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.098086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.098983 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.098987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.099620 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.099621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.099703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.099707 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.099837 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.099874 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260808 02:22:35.100238 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cb7c3c883045db91e2642a0ba3fbc7" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9" is_pre_election: true
I20260808 02:22:35.100378 998 raft_consensus.cc:2454] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 0.
I20260808 02:22:35.100625 1070 leader_election.cc:302] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375ce3ceefd24c278478525e12b8fdb9, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:35.100726 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cb7c3c883045db91e2642a0ba3fbc7" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" is_pre_election: true
I20260808 02:22:35.100790 1217 raft_consensus.cc:2781] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:35.100842 1217 raft_consensus.cc:482] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:35.100873 1217 raft_consensus.cc:3037] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.100859 854 raft_consensus.cc:2454] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 0.
I20260808 02:22:35.101610 1217 raft_consensus.cc:504] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.101907 1217 leader_election.cc:288] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 election: Requested vote from peers 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247), 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087)
I20260808 02:22:35.101920 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cb7c3c883045db91e2642a0ba3fbc7" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9"
I20260808 02:22:35.102005 998 raft_consensus.cc:3037] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.102006 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cb7c3c883045db91e2642a0ba3fbc7" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c"
I20260808 02:22:35.102061 854 raft_consensus.cc:3037] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.102849 998 raft_consensus.cc:2454] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 1.
I20260808 02:22:35.102859 854 raft_consensus.cc:2454] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 1.
I20260808 02:22:35.103055 1070 leader_election.cc:302] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375ce3ceefd24c278478525e12b8fdb9, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:35.103158 1217 raft_consensus.cc:2781] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:35.103294 1217 raft_consensus.cc:686] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 1 LEADER]: Becoming Leader. State: Replica: e45a7827306e497b827b855eba7a5227, State: Running, Role: LEADER
I20260808 02:22:35.103394 1217 consensus_queue.cc:237] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } }
I20260808 02:22:35.104328 693 catalog_manager.cc:5495] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 reported cstate change: term changed from 0 to 1, leader changed from <none> to e45a7827306e497b827b855eba7a5227 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } health_report { overall_health: UNKNOWN } } }
W20260808 02:22:35.113931 1046 tablet.cc:2369] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:22:35 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155754, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:35.143201 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.136629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34919 (local address 127.0.0.1:57676)
0808 02:22:35.136829 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.136841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.136924 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.137055 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.137056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.137453 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:35.137501 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.137503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.138214 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.138215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.138739 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.138741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.139048 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.139049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.139091 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.139583 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:35.139588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:35.140965 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:35.142284 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:35.142291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:35.142293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:35.142491 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:35.142660 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:35.142661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:35.142661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:35.142716 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:35.142927 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:35.142933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:35.142972 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.143023 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.143109 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":117,"threads_started":1}
I20260808 02:22:35.143276 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.136690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.30.160.62:34919)
0808 02:22:35.136749 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.136751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.136910 (+ 159us) server_negotiation.cc:407] Connection header received
0808 02:22:35.136968 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.136969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.136978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.137038 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:35.137570 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.138121 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.138805 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.138984 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.139033 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.141050 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:35.141051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:35.141051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:35.141057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:35.142214 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.142555 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.142556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.142556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.142603 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:35.142769 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:35.142769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:35.142770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:35.142873 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:35.142922 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.143055 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.143153 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":35}
I20260808 02:22:35.145388 693 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:57676:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:22:35.145622 693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:35.147069 976 tablet_service.cc:1467] Processing CreateTablet for tablet 918dc04b91b24690992fc87932d94fc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d5fcd2130364c08a0c31deb67acb50f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:22:35.147091 1120 tablet_service.cc:1467] Processing CreateTablet for tablet 918dc04b91b24690992fc87932d94fc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d5fcd2130364c08a0c31deb67acb50f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:22:35.147082 833 tablet_service.cc:1467] Processing CreateTablet for tablet 918dc04b91b24690992fc87932d94fc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d5fcd2130364c08a0c31deb67acb50f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:22:35.147244 1120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918dc04b91b24690992fc87932d94fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.147244 833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918dc04b91b24690992fc87932d94fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.147244 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918dc04b91b24690992fc87932d94fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.148512 1212 tablet_bootstrap.cc:492] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: Bootstrap starting.
I20260808 02:22:35.148900 1211 tablet_bootstrap.cc:492] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: Bootstrap starting.
I20260808 02:22:35.149266 1210 tablet_bootstrap.cc:492] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: Bootstrap starting.
I20260808 02:22:35.149283 1212 tablet_bootstrap.cc:654] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.149617 1211 tablet_bootstrap.cc:654] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.149704 1210 tablet_bootstrap.cc:654] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.150068 1212 tablet_bootstrap.cc:492] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: No bootstrap required, opened a new log
I20260808 02:22:35.150146 1212 ts_tablet_manager.cc:1397] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:22:35.150341 1211 tablet_bootstrap.cc:492] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: No bootstrap required, opened a new log
I20260808 02:22:35.150326 1212 raft_consensus.cc:348] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.150388 1211 ts_tablet_manager.cc:1397] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:22:35.150399 1212 raft_consensus.cc:374] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.150422 1212 raft_consensus.cc:729] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375ce3ceefd24c278478525e12b8fdb9, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.150485 1212 consensus_queue.cc:260] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.150535 1211 raft_consensus.cc:348] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.150586 1212 ts_tablet_manager.cc:1428] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:35.150604 1211 raft_consensus.cc:374] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.150627 1211 raft_consensus.cc:729] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.150671 1211 consensus_queue.cc:260] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.150753 1211 ts_tablet_manager.cc:1428] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:35.150677 1218 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.150794 1218 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:35.150811 1216 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.150913 1216 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:35.151105 1210 tablet_bootstrap.cc:492] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: No bootstrap required, opened a new log
I20260808 02:22:35.151176 1210 ts_tablet_manager.cc:1397] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:35.151331 1210 raft_consensus.cc:348] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.151417 1210 raft_consensus.cc:374] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.151443 1210 raft_consensus.cc:729] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45a7827306e497b827b855eba7a5227, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.151496 1210 consensus_queue.cc:260] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.151590 1210 ts_tablet_manager.cc:1428] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:35.151629 1217 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.151697 1217 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 02:22:35.194895 903 tablet.cc:2369] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:35.205941 1217 raft_consensus.cc:482] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:35.206022 1217 raft_consensus.cc:504] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.206182 1217 leader_election.cc:288] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087), 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247)
I20260808 02:22:35.206444 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918dc04b91b24690992fc87932d94fc1" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9" is_pre_election: true
I20260808 02:22:35.206440 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918dc04b91b24690992fc87932d94fc1" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" is_pre_election: true
I20260808 02:22:35.206547 854 raft_consensus.cc:2454] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 0.
I20260808 02:22:35.206547 998 raft_consensus.cc:2454] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 0.
I20260808 02:22:35.206725 1072 leader_election.cc:302] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18a884945bbe4cbaa058a1d3c5e27f3c, 375ce3ceefd24c278478525e12b8fdb9, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:35.206848 1217 raft_consensus.cc:2781] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:35.206918 1217 raft_consensus.cc:482] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:35.206946 1217 raft_consensus.cc:3037] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.207644 1217 raft_consensus.cc:504] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.207790 1217 leader_election.cc:288] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 election: Requested vote from peers 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087), 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247)
I20260808 02:22:35.207970 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918dc04b91b24690992fc87932d94fc1" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c"
I20260808 02:22:35.208006 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918dc04b91b24690992fc87932d94fc1" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9"
I20260808 02:22:35.208065 998 raft_consensus.cc:3037] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.208070 854 raft_consensus.cc:3037] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.208714 998 raft_consensus.cc:2454] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 1.
I20260808 02:22:35.208724 854 raft_consensus.cc:2454] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 1.
I20260808 02:22:35.208887 1070 leader_election.cc:302] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375ce3ceefd24c278478525e12b8fdb9, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:35.208994 1217 raft_consensus.cc:2781] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:35.209059 1217 raft_consensus.cc:686] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [term 1 LEADER]: Becoming Leader. State: Replica: e45a7827306e497b827b855eba7a5227, State: Running, Role: LEADER
I20260808 02:22:35.209102 1217 consensus_queue.cc:237] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.209414 1220 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: TxnStatusTablet state changed. Reason: New leader e45a7827306e497b827b855eba7a5227. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.209507 1220 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:22:35.209726 1232 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:22:35.209738 692 catalog_manager.cc:5495] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 reported cstate change: term changed from 0 to 1, leader changed from <none> to e45a7827306e497b827b855eba7a5227 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } health_report { overall_health: HEALTHY } } }
I20260808 02:22:35.209862 1232 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:22:35.226797 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.224097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.30.160.3:38905)
0808 02:22:35.224160 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.224162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.224172 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:35.224244 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.224245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.224261 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.224320 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:35.224463 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.224947 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.226065 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.226246 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.226298 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.226511 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.226599 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.226652 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.226726 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.226746 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:22:35.226858 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.224015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:38905 (local address 127.0.0.1:58206)
0808 02:22:35.224092 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.224106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.224159 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.224344 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.224345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.224346 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:35.224399 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.224402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.225025 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.225026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.225990 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.225994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.226326 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.226328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.226399 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.226418 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.226658 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.226667 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.226724 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.226775 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 02:22:35.229444 998 raft_consensus.cc:1261] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Refusing update from remote peer e45a7827306e497b827b855eba7a5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:35.229458 854 raft_consensus.cc:1261] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Refusing update from remote peer e45a7827306e497b827b855eba7a5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:35.229768 1220 consensus_queue.cc:1035] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:35.229897 1217 consensus_queue.cc:1035] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:35.230950 1216 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: TxnStatusTablet state changed. Reason: New leader e45a7827306e497b827b855eba7a5227. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.231045 1216 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:35.231122 1218 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: TxnStatusTablet state changed. Reason: New leader e45a7827306e497b827b855eba7a5227. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.231230 1218 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:35.232307 1217 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.232411 1217 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:22:35.232580 1220 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.232640 1220 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P e45a7827306e497b827b855eba7a5227: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:22:35.232975 1237 mvcc.cc:205] Tried to move back new op lower bound from 7316093973417414656 to 7316093973336813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316093973417414656}]
I20260808 02:22:35.233182 1218 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.233266 1218 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P 375ce3ceefd24c278478525e12b8fdb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:35.233275 1216 tablet_replica.cc:439] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } } }
I20260808 02:22:35.233372 1216 tablet_replica.cc:442] T 918dc04b91b24690992fc87932d94fc1 P 18a884945bbe4cbaa058a1d3c5e27f3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:22:35.233525 1235 mvcc.cc:205] Tried to move back new op lower bound from 7316093973417414656 to 7316093973336813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316093973417414656 or (T in {7316093973417414656})}]
I20260808 02:22:35.233685 1236 mvcc.cc:205] Tried to move back new op lower bound from 7316093973417414656 to 7316093973336813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316093973417414656}]
I20260808 02:22:35.238678 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.235818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.30.160.3:38905)
0808 02:22:35.235880 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.235882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.235920 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:22:35.235945 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.235946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.235957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.236018 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:35.236158 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.236714 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.237855 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.238052 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.238144 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.238386 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.238424 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.238490 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.238560 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.238583 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:22:35.244405 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.241381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:38905 (local address 127.30.160.3:55967)
0808 02:22:35.241486 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.241499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.241542 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.241670 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.241670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.241671 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:35.241721 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.241723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.242577 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.242578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.243252 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.243254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.243969 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.243970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.244041 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.244055 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.244268 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.244269 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.244307 (+ 38us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.244341 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:22:35.244415 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.241481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:55967 (local address 127.30.160.3:38905)
0808 02:22:35.241533 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.241535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.241545 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:35.241599 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.241600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.241610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.241670 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:35.241782 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.242496 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.243459 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.243776 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.243858 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.244160 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.244193 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.244254 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.244309 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.244327 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:22:35.246892 854 raft_consensus.cc:1261] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Refusing update from remote peer e45a7827306e497b827b855eba7a5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:35.246915 998 raft_consensus.cc:1261] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Refusing update from remote peer e45a7827306e497b827b855eba7a5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:35.247113 1246 consensus_queue.cc:1035] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:35.247218 1220 consensus_queue.cc:1035] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:35.262380 692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:35.262493 692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:35.263226 692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Sending DeleteTablet for 3 replicas of tablet a7cb7c3c883045db91e2642a0ba3fbc7
I20260808 02:22:35.263520 976 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cb7c3c883045db91e2642a0ba3fbc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:35 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:38574
I20260808 02:22:35.263551 833 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cb7c3c883045db91e2642a0ba3fbc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:35 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:49510
I20260808 02:22:35.263674 1119 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cb7c3c883045db91e2642a0ba3fbc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:35 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:58190
I20260808 02:22:35.263816 1250 tablet_replica.cc:330] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: stopping tablet replica
I20260808 02:22:35.263948 1250 raft_consensus.cc:2229] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:35.263994 1252 tablet_replica.cc:330] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: stopping tablet replica
I20260808 02:22:35.264127 1250 raft_consensus.cc:2258] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:35.264233 1252 raft_consensus.cc:2229] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:35.264216 692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:35.264405 692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:35.264568 1252 raft_consensus.cc:2258] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:35.265090 1252 ts_tablet_manager.cc:1905] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:35.265393 1250 ts_tablet_manager.cc:1905] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:35.265697 833 tablet_service.cc:1467] Processing CreateTablet for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 (DEFAULT_TABLE table=test-table [id=c667e738dfce4609b7d52f9aa43ace64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:35.265766 976 tablet_service.cc:1467] Processing CreateTablet for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 (DEFAULT_TABLE table=test-table [id=c667e738dfce4609b7d52f9aa43ace64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:35.265826 833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.265897 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.267167 1211 tablet_bootstrap.cc:492] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Bootstrap starting.
I20260808 02:22:35.267302 1212 tablet_bootstrap.cc:492] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Bootstrap starting.
I20260808 02:22:35.267289 1119 tablet_service.cc:1467] Processing CreateTablet for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 (DEFAULT_TABLE table=test-table [id=c667e738dfce4609b7d52f9aa43ace64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:35.267405 1119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:35.267410 1250 ts_tablet_manager.cc:1918] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:35.267485 1250 log.cc:1196] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/a7cb7c3c883045db91e2642a0ba3fbc7
I20260808 02:22:35.267658 1211 tablet_bootstrap.cc:654] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.267712 1212 tablet_bootstrap.cc:654] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.267792 1250 ts_tablet_manager.cc:1939] T a7cb7c3c883045db91e2642a0ba3fbc7 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting consensus metadata
I20260808 02:22:35.268476 1212 tablet_bootstrap.cc:492] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: No bootstrap required, opened a new log
I20260808 02:22:35.268519 1212 ts_tablet_manager.cc:1397] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:35.268648 1212 raft_consensus.cc:348] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.268743 1212 raft_consensus.cc:374] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.268764 1212 raft_consensus.cc:729] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375ce3ceefd24c278478525e12b8fdb9, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.268810 1212 consensus_queue.cc:260] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.268893 1212 ts_tablet_manager.cc:1428] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:35.269047 1210 tablet_bootstrap.cc:492] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Bootstrap starting.
I20260808 02:22:35.269486 1210 tablet_bootstrap.cc:654] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:35.269778 680 catalog_manager.cc:4841] TS 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087): tablet a7cb7c3c883045db91e2642a0ba3fbc7 (table test-table [id=9a707be4f5bb4c8c8da7da7b452b37d8]) successfully deleted
I20260808 02:22:35.270571 1210 tablet_bootstrap.cc:492] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: No bootstrap required, opened a new log
I20260808 02:22:35.270634 1210 ts_tablet_manager.cc:1397] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:35.270804 1210 raft_consensus.cc:348] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.270872 1210 raft_consensus.cc:374] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.270895 1210 raft_consensus.cc:729] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45a7827306e497b827b855eba7a5227, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.270946 1210 consensus_queue.cc:260] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.271068 1210 ts_tablet_manager.cc:1428] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:35.271382 1252 ts_tablet_manager.cc:1918] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:35.271476 1252 log.cc:1196] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/a7cb7c3c883045db91e2642a0ba3fbc7
I20260808 02:22:35.271771 1252 ts_tablet_manager.cc:1939] T a7cb7c3c883045db91e2642a0ba3fbc7 P e45a7827306e497b827b855eba7a5227: Deleting consensus metadata
I20260808 02:22:35.272207 681 catalog_manager.cc:4841] TS e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905): tablet a7cb7c3c883045db91e2642a0ba3fbc7 (table test-table [id=9a707be4f5bb4c8c8da7da7b452b37d8]) successfully deleted
I20260808 02:22:35.276011 1251 tablet_replica.cc:330] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: stopping tablet replica
I20260808 02:22:35.276146 1251 raft_consensus.cc:2229] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:35.276245 1251 raft_consensus.cc:2258] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:35.276340 1211 tablet_bootstrap.cc:492] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: No bootstrap required, opened a new log
I20260808 02:22:35.276397 1211 ts_tablet_manager.cc:1397] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260808 02:22:35.276546 1211 raft_consensus.cc:348] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.276603 1211 raft_consensus.cc:374] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:35.276616 1211 raft_consensus.cc:729] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Initialized, Role: FOLLOWER
I20260808 02:22:35.276671 1211 consensus_queue.cc:260] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.276768 1211 ts_tablet_manager.cc:1428] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:35.277186 1251 ts_tablet_manager.cc:1905] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:35.278745 1251 ts_tablet_manager.cc:1918] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:35.278782 1216 raft_consensus.cc:482] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:35.278836 1251 log.cc:1196] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/a7cb7c3c883045db91e2642a0ba3fbc7
I20260808 02:22:35.278832 1216 raft_consensus.cc:504] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.279098 1251 ts_tablet_manager.cc:1939] T a7cb7c3c883045db91e2642a0ba3fbc7 P 375ce3ceefd24c278478525e12b8fdb9: Deleting consensus metadata
I20260808 02:22:35.279132 1216 leader_election.cc:288] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247), e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905)
I20260808 02:22:35.279528 678 catalog_manager.cc:4841] TS 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247): tablet a7cb7c3c883045db91e2642a0ba3fbc7 (table test-table [id=9a707be4f5bb4c8c8da7da7b452b37d8]) successfully deleted
I20260808 02:22:35.282904 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.279756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:40805 (local address 127.30.160.3:38905)
0808 02:22:35.279802 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.279804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.279822 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:22:35.279880 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.279880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.279906 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.279952 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.280176 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.280908 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.282099 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.282497 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.282569 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.282582 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.282795 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.282847 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:22:35.282903 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.279691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:39139 (local address 127.30.160.2:45247)
0808 02:22:35.279743 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.279745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.279753 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:35.279775 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.279776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.279784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.279834 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:35.280056 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.280579 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.281938 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.282509 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.282580 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.282588 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.282796 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.282839 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 02:22:35.283042 1188 tablet.cc:2369] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:35.282913 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.279341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:45247 (local address 127.30.160.1:39139)
0808 02:22:35.279457 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.279471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.279530 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.279882 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.279883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.279885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.279976 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.279978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.280668 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.280669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.281841 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.281844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.282626 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.282628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.282732 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.282736 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.282791 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.282834 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260808 02:22:35.283176 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.279522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:38905 (local address 127.30.160.1:40805)
0808 02:22:35.279757 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.279767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.279824 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.280001 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.280002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.280004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:35.280093 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.280094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.280999 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.281001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.282016 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.282018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.282618 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.282619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.282706 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.282710 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.283020 (+ 310us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.283066 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
I20260808 02:22:35.283378 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e42da3e04a64cd1a6dbc7b0cbb816e3" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9" is_pre_election: true
I20260808 02:22:35.283459 998 raft_consensus.cc:2454] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 0.
I20260808 02:22:35.283605 783 leader_election.cc:302] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a884945bbe4cbaa058a1d3c5e27f3c, 375ce3ceefd24c278478525e12b8fdb9; no voters:
I20260808 02:22:35.283607 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e42da3e04a64cd1a6dbc7b0cbb816e3" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45a7827306e497b827b855eba7a5227" is_pre_election: true
I20260808 02:22:35.283713 1141 raft_consensus.cc:2454] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 0.
I20260808 02:22:35.283718 1216 raft_consensus.cc:2781] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:35.283746 1216 raft_consensus.cc:482] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:35.283764 1216 raft_consensus.cc:3037] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.284575 1216 raft_consensus.cc:504] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.284722 1216 leader_election.cc:288] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 election: Requested vote from peers 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247), e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905)
I20260808 02:22:35.284912 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e42da3e04a64cd1a6dbc7b0cbb816e3" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9"
I20260808 02:22:35.284988 998 raft_consensus.cc:3037] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.284976 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e42da3e04a64cd1a6dbc7b0cbb816e3" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45a7827306e497b827b855eba7a5227"
I20260808 02:22:35.285048 1141 raft_consensus.cc:3037] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:35.285519 998 raft_consensus.cc:2454] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 1.
I20260808 02:22:35.285684 783 leader_election.cc:302] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a884945bbe4cbaa058a1d3c5e27f3c, 375ce3ceefd24c278478525e12b8fdb9; no voters:
I20260808 02:22:35.285838 1141 raft_consensus.cc:2454] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 1.
I20260808 02:22:35.285941 1216 raft_consensus.cc:2781] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:35.286085 1216 raft_consensus.cc:686] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 LEADER]: Becoming Leader. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Running, Role: LEADER
I20260808 02:22:35.286166 1216 consensus_queue.cc:237] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:35.286739 692 catalog_manager.cc:5495] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1). New cstate: current_term: 1 leader_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:35.293588 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.291186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.30.160.1:38087)
0808 02:22:35.291243 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.291245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.291254 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:35.291308 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.291308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.291327 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.291409 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:35.291514 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.292050 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.292784 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.292990 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.293074 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.293240 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.293351 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.293429 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.293487 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.293508 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:22:35.298724 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.296076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:38905 (local address 127.30.160.1:60665)
0808 02:22:35.296240 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.296250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.296320 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.296467 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.296468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.296469 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:35.296519 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.296520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.297355 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.297356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.297892 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.297893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.298363 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.298364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.298407 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.298420 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.298592 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.298593 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.298628 (+ 35us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.298658 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260808 02:22:35.298727 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.296127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:60665 (local address 127.30.160.3:38905)
0808 02:22:35.296171 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.296174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.296351 (+ 177us) server_negotiation.cc:407] Connection header received
0808 02:22:35.296382 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.296383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.296395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.296465 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:35.296577 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.297270 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.297971 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.298264 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.298335 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.298497 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.298530 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.298585 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.298631 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.298650 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:35.301081 998 raft_consensus.cc:1261] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Refusing update from remote peer 18a884945bbe4cbaa058a1d3c5e27f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:35.301312 1141 raft_consensus.cc:1261] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Refusing update from remote peer 18a884945bbe4cbaa058a1d3c5e27f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:35.301543 1259 consensus_queue.cc:1035] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:35.301677 1216 consensus_queue.cc:1035] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:35.309429 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.306536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:38087 (local address 127.30.160.3:44625)
0808 02:22:35.306589 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:35.306599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:35.306636 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:35.306958 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:35.306960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:35.306961 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:22:35.307001 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.307003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.307731 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.307732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.308601 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:35.308605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.309089 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.309090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:35.309140 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.309144 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.309321 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.309322 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:22:35.309349 (+ 27us) client_negotiation.cc:240] Negotiation successful
0808 02:22:35.309379 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:22:35.309448 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:35.306732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:44625 (local address 127.30.160.1:38087)
0808 02:22:35.306776 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:35.306778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:35.306786 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:35.306805 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:35.306805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:35.306822 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:35.306876 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:35.307078 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.307598 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.308725 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:35.309000 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:35.309078 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:35.309222 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:35.309255 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:35.309317 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:35.309364 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 02:22:35.309383 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 02:22:35.309947 1236 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:22:36.834821 693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:36.834920 693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:36.835809 693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Sending DeleteTablet for 3 replicas of tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3
I20260808 02:22:36.836133 833 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:36 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:49510
I20260808 02:22:36.836180 1119 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:36 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:58190
I20260808 02:22:36.836380 976 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:36 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:38574
I20260808 02:22:36.836609 1295 tablet_replica.cc:330] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: stopping tablet replica
I20260808 02:22:36.836683 1295 raft_consensus.cc:2229] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:36.836721 1295 raft_consensus.cc:2258] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:36.836958 1295 ts_tablet_manager.cc:1905] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:36.838510 1295 ts_tablet_manager.cc:1918] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:22:36.838596 1295 log.cc:1196] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/6e42da3e04a64cd1a6dbc7b0cbb816e3
I20260808 02:22:36.838856 1295 ts_tablet_manager.cc:1939] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 375ce3ceefd24c278478525e12b8fdb9: Deleting consensus metadata
I20260808 02:22:36.839219 678 catalog_manager.cc:4841] TS 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247): tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 (table test-table [id=c667e738dfce4609b7d52f9aa43ace64]) successfully deleted
I20260808 02:22:36.839484 1294 tablet_replica.cc:330] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: stopping tablet replica
I20260808 02:22:36.839555 1294 raft_consensus.cc:2229] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:36.839650 1294 raft_consensus.cc:2258] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:36.839794 1293 tablet_replica.cc:330] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: stopping tablet replica
I20260808 02:22:36.839861 1293 raft_consensus.cc:2229] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:36.839948 1293 raft_consensus.cc:2258] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:36.839946 1294 ts_tablet_manager.cc:1905] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:36.840178 1293 ts_tablet_manager.cc:1905] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:36.841634 1293 ts_tablet_manager.cc:1918] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:22:36.841912 1293 log.cc:1196] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/6e42da3e04a64cd1a6dbc7b0cbb816e3
I20260808 02:22:36.842150 1293 ts_tablet_manager.cc:1939] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P e45a7827306e497b827b855eba7a5227: Deleting consensus metadata
I20260808 02:22:36.842360 1294 ts_tablet_manager.cc:1918] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:22:36.842474 1294 log.cc:1196] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/6e42da3e04a64cd1a6dbc7b0cbb816e3
I20260808 02:22:36.842495 681 catalog_manager.cc:4841] TS e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905): tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 (table test-table [id=c667e738dfce4609b7d52f9aa43ace64]) successfully deleted
I20260808 02:22:36.842784 1294 ts_tablet_manager.cc:1939] T 6e42da3e04a64cd1a6dbc7b0cbb816e3 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting consensus metadata
I20260808 02:22:36.843111 680 catalog_manager.cc:4841] TS 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087): tablet 6e42da3e04a64cd1a6dbc7b0cbb816e3 (table test-table [id=c667e738dfce4609b7d52f9aa43ace64]) successfully deleted
Aug 08 02:22:36 dist-test-slave-zkpd krb5kdc[649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:22:36 dist-test-slave-zkpd krb5kdc[649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:36.861605 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:36.854461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.30.160.62:34919)
0808 02:22:36.854608 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:36.854610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:36.854633 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:22:36.854719 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:36.854720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:36.854731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:36.854804 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:36.855363 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:36.855984 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:36.856698 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:36.856927 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:36.856981 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:36.859214 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:36.859215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:36.859215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:36.859221 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:36.860657 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:36.860966 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:36.860966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:36.860967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:36.861019 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:36.861183 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:36.861184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:36.861184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:36.861283 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:36.861370 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:36.861470 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 02:22:36.861533 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260808 02:22:36.863181 693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:36.863327 693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:36.864727 833 tablet_service.cc:1467] Processing CreateTablet for tablet c9a83c8d9d2347a283a2baba8453e043 (DEFAULT_TABLE table=test-table [id=bb5b4a05308b4570baffec9f81a8d368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:36.864765 1119 tablet_service.cc:1467] Processing CreateTablet for tablet c9a83c8d9d2347a283a2baba8453e043 (DEFAULT_TABLE table=test-table [id=bb5b4a05308b4570baffec9f81a8d368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:36.864812 976 tablet_service.cc:1467] Processing CreateTablet for tablet c9a83c8d9d2347a283a2baba8453e043 (DEFAULT_TABLE table=test-table [id=bb5b4a05308b4570baffec9f81a8d368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:36.864867 833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a83c8d9d2347a283a2baba8453e043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:36.864900 1119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a83c8d9d2347a283a2baba8453e043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:36.864930 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a83c8d9d2347a283a2baba8453e043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:36.866010 1305 tablet_bootstrap.cc:492] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Bootstrap starting.
I20260808 02:22:36.866428 1305 tablet_bootstrap.cc:654] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:36.866451 1306 tablet_bootstrap.cc:492] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Bootstrap starting.
I20260808 02:22:36.866468 1307 tablet_bootstrap.cc:492] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Bootstrap starting.
I20260808 02:22:36.866873 1306 tablet_bootstrap.cc:654] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:36.866984 1307 tablet_bootstrap.cc:654] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:36.867244 1305 tablet_bootstrap.cc:492] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: No bootstrap required, opened a new log
I20260808 02:22:36.867336 1305 ts_tablet_manager.cc:1397] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:36.867453 1306 tablet_bootstrap.cc:492] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: No bootstrap required, opened a new log
I20260808 02:22:36.867558 1306 ts_tablet_manager.cc:1397] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:36.867523 1305 raft_consensus.cc:348] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:36.867599 1305 raft_consensus.cc:374] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:36.867614 1305 raft_consensus.cc:729] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375ce3ceefd24c278478525e12b8fdb9, State: Initialized, Role: FOLLOWER
I20260808 02:22:36.867647 1307 tablet_bootstrap.cc:492] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: No bootstrap required, opened a new log
I20260808 02:22:36.867691 1307 ts_tablet_manager.cc:1397] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:36.867681 1305 consensus_queue.cc:260] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:36.867820 1306 raft_consensus.cc:348] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:36.867828 1307 raft_consensus.cc:348] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:36.867895 1305 ts_tablet_manager.cc:1428] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:36.867949 1307 raft_consensus.cc:374] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:36.867908 1306 raft_consensus.cc:374] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:36.867977 1307 raft_consensus.cc:729] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45a7827306e497b827b855eba7a5227, State: Initialized, Role: FOLLOWER
I20260808 02:22:36.867998 1306 raft_consensus.cc:729] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Initialized, Role: FOLLOWER
I20260808 02:22:36.868024 1307 consensus_queue.cc:260] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:36.868058 1306 consensus_queue.cc:260] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:36.868132 1307 ts_tablet_manager.cc:1428] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:36.868600 1306 ts_tablet_manager.cc:1428] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:37.175001 1287 raft_consensus.cc:482] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:37.175127 1287 raft_consensus.cc:504] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.175379 1287 leader_election.cc:288] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247), e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905)
I20260808 02:22:37.175674 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a83c8d9d2347a283a2baba8453e043" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45a7827306e497b827b855eba7a5227" is_pre_election: true
I20260808 02:22:37.175673 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a83c8d9d2347a283a2baba8453e043" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9" is_pre_election: true
I20260808 02:22:37.175793 998 raft_consensus.cc:2454] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 0.
I20260808 02:22:37.175793 1141 raft_consensus.cc:2454] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 0.
I20260808 02:22:37.176033 786 leader_election.cc:302] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a884945bbe4cbaa058a1d3c5e27f3c, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:37.176168 1287 raft_consensus.cc:2781] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:37.176242 1287 raft_consensus.cc:482] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:37.176270 1287 raft_consensus.cc:3037] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:37.177004 1287 raft_consensus.cc:504] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.177157 1287 leader_election.cc:288] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 election: Requested vote from peers 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247), e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905)
I20260808 02:22:37.177328 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a83c8d9d2347a283a2baba8453e043" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9"
I20260808 02:22:37.177423 998 raft_consensus.cc:3037] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:37.177367 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a83c8d9d2347a283a2baba8453e043" candidate_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45a7827306e497b827b855eba7a5227"
I20260808 02:22:37.177459 1141 raft_consensus.cc:3037] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:37.178079 998 raft_consensus.cc:2454] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 1.
I20260808 02:22:37.178083 1141 raft_consensus.cc:2454] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a884945bbe4cbaa058a1d3c5e27f3c in term 1.
I20260808 02:22:37.178269 786 leader_election.cc:302] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a884945bbe4cbaa058a1d3c5e27f3c, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:37.178397 1287 raft_consensus.cc:2781] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:37.178490 1287 raft_consensus.cc:686] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 LEADER]: Becoming Leader. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Running, Role: LEADER
I20260808 02:22:37.178573 1287 consensus_queue.cc:237] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.179396 694 catalog_manager.cc:5495] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1). New cstate: current_term: 1 leader_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:37.223192 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:37.220853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.30.160.1:38087)
0808 02:22:37.221033 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:37.221035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:37.221042 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:37.221063 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:37.221063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:37.221073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:37.221130 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:37.221277 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:37.221756 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:37.222559 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:37.222763 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:37.222824 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:37.222963 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:37.222990 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:37.223028 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:37.223074 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 02:22:37.223092 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
I20260808 02:22:37.225603 1141 raft_consensus.cc:1261] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Refusing update from remote peer 18a884945bbe4cbaa058a1d3c5e27f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:37.225737 998 raft_consensus.cc:1261] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Refusing update from remote peer 18a884945bbe4cbaa058a1d3c5e27f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:37.225862 1268 consensus_queue.cc:1035] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:37.226034 1268 consensus_queue.cc:1035] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:37.236068 692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:37.236166 692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:37.236729 692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Sending DeleteTablet for 3 replicas of tablet c9a83c8d9d2347a283a2baba8453e043
I20260808 02:22:37.236969 833 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a83c8d9d2347a283a2baba8453e043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:37 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:49510
I20260808 02:22:37.237003 976 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a83c8d9d2347a283a2baba8453e043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:37 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:38574
I20260808 02:22:37.237061 1294 tablet_replica.cc:330] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: stopping tablet replica
I20260808 02:22:37.237106 1294 raft_consensus.cc:2229] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:37.237098 1119 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a83c8d9d2347a283a2baba8453e043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:37 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:58190
I20260808 02:22:37.237100 1295 tablet_replica.cc:330] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: stopping tablet replica
I20260808 02:22:37.237159 1294 raft_consensus.cc:2258] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:37.237172 1293 tablet_replica.cc:330] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: stopping tablet replica
I20260808 02:22:37.237169 1295 raft_consensus.cc:2229] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:37.237223 1293 raft_consensus.cc:2229] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:37.237236 1295 raft_consensus.cc:2258] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:37.237268 1293 raft_consensus.cc:2258] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:37.237569 1295 ts_tablet_manager.cc:1905] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:37.237591 1294 ts_tablet_manager.cc:1905] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:37.238840 1293 ts_tablet_manager.cc:1905] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:37.238838 692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:37.239048 692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:37.239094 1294 ts_tablet_manager.cc:1918] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:37.239159 1294 log.cc:1196] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/c9a83c8d9d2347a283a2baba8453e043
I20260808 02:22:37.239259 1295 ts_tablet_manager.cc:1918] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:37.239332 1295 log.cc:1196] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/c9a83c8d9d2347a283a2baba8453e043
I20260808 02:22:37.239408 1294 ts_tablet_manager.cc:1939] T c9a83c8d9d2347a283a2baba8453e043 P 18a884945bbe4cbaa058a1d3c5e27f3c: Deleting consensus metadata
I20260808 02:22:37.239557 1295 ts_tablet_manager.cc:1939] T c9a83c8d9d2347a283a2baba8453e043 P 375ce3ceefd24c278478525e12b8fdb9: Deleting consensus metadata
I20260808 02:22:37.239715 680 catalog_manager.cc:4841] TS 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087): tablet c9a83c8d9d2347a283a2baba8453e043 (table test-table [id=bb5b4a05308b4570baffec9f81a8d368]) successfully deleted
I20260808 02:22:37.239991 678 catalog_manager.cc:4841] TS 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247): tablet c9a83c8d9d2347a283a2baba8453e043 (table test-table [id=bb5b4a05308b4570baffec9f81a8d368]) successfully deleted
I20260808 02:22:37.240581 1293 ts_tablet_manager.cc:1918] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:22:37.240640 1293 log.cc:1196] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/c9a83c8d9d2347a283a2baba8453e043
I20260808 02:22:37.240895 1293 ts_tablet_manager.cc:1939] T c9a83c8d9d2347a283a2baba8453e043 P e45a7827306e497b827b855eba7a5227: Deleting consensus metadata
I20260808 02:22:37.240983 833 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa407ee07346dc87520ee5465ff408 (DEFAULT_TABLE table=test-table [id=91935e5719504f248053cb1e3294e9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:37.241022 1119 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa407ee07346dc87520ee5465ff408 (DEFAULT_TABLE table=test-table [id=91935e5719504f248053cb1e3294e9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:37.241083 833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25aa407ee07346dc87520ee5465ff408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:37.241159 1119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25aa407ee07346dc87520ee5465ff408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:37.241233 681 catalog_manager.cc:4841] TS e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905): tablet c9a83c8d9d2347a283a2baba8453e043 (table test-table [id=bb5b4a05308b4570baffec9f81a8d368]) successfully deleted
I20260808 02:22:37.242036 976 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa407ee07346dc87520ee5465ff408 (DEFAULT_TABLE table=test-table [id=91935e5719504f248053cb1e3294e9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:37.242234 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25aa407ee07346dc87520ee5465ff408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:37.241981 1306 tablet_bootstrap.cc:492] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c: Bootstrap starting.
I20260808 02:22:37.242374 1307 tablet_bootstrap.cc:492] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Bootstrap starting.
I20260808 02:22:37.242658 1306 tablet_bootstrap.cc:654] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:37.242681 1307 tablet_bootstrap.cc:654] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:37.243283 1306 tablet_bootstrap.cc:492] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c: No bootstrap required, opened a new log
I20260808 02:22:37.243356 1306 ts_tablet_manager.cc:1397] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:37.243372 1305 tablet_bootstrap.cc:492] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Bootstrap starting.
I20260808 02:22:37.243486 1306 raft_consensus.cc:348] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.243559 1306 raft_consensus.cc:374] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:37.243575 1306 raft_consensus.cc:729] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a884945bbe4cbaa058a1d3c5e27f3c, State: Initialized, Role: FOLLOWER
I20260808 02:22:37.243633 1306 consensus_queue.cc:260] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.243669 1305 tablet_bootstrap.cc:654] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:37.243731 1306 ts_tablet_manager.cc:1428] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:37.244216 1307 tablet_bootstrap.cc:492] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: No bootstrap required, opened a new log
I20260808 02:22:37.244280 1307 ts_tablet_manager.cc:1397] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:22:37.244454 1305 tablet_bootstrap.cc:492] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: No bootstrap required, opened a new log
I20260808 02:22:37.244433 1307 raft_consensus.cc:348] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.244505 1307 raft_consensus.cc:374] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:37.244509 1305 ts_tablet_manager.cc:1397] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:37.244532 1307 raft_consensus.cc:729] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45a7827306e497b827b855eba7a5227, State: Initialized, Role: FOLLOWER
I20260808 02:22:37.244591 1307 consensus_queue.cc:260] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.244700 1307 ts_tablet_manager.cc:1428] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:37.244670 1305 raft_consensus.cc:348] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.244757 1305 raft_consensus.cc:374] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:37.244796 1305 raft_consensus.cc:729] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375ce3ceefd24c278478525e12b8fdb9, State: Initialized, Role: FOLLOWER
I20260808 02:22:37.244853 1305 consensus_queue.cc:260] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.244975 1305 ts_tablet_manager.cc:1428] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:37.595455 1269 raft_consensus.cc:482] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:37.595563 1269 raft_consensus.cc:504] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.595760 1269 leader_election.cc:288] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087), 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247)
I20260808 02:22:37.595990 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25aa407ee07346dc87520ee5465ff408" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9" is_pre_election: true
I20260808 02:22:37.595996 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25aa407ee07346dc87520ee5465ff408" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" is_pre_election: true
I20260808 02:22:37.596102 854 raft_consensus.cc:2454] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 0.
I20260808 02:22:37.596107 998 raft_consensus.cc:2454] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 0.
I20260808 02:22:37.596271 1070 leader_election.cc:302] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375ce3ceefd24c278478525e12b8fdb9, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:37.596398 1269 raft_consensus.cc:2781] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:37.596450 1269 raft_consensus.cc:482] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:37.596472 1269 raft_consensus.cc:3037] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:37.597198 1269 raft_consensus.cc:504] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.597340 1269 leader_election.cc:288] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 election: Requested vote from peers 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087), 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247)
I20260808 02:22:37.597465 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25aa407ee07346dc87520ee5465ff408" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c"
I20260808 02:22:37.597496 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25aa407ee07346dc87520ee5465ff408" candidate_uuid: "e45a7827306e497b827b855eba7a5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375ce3ceefd24c278478525e12b8fdb9"
I20260808 02:22:37.597568 998 raft_consensus.cc:3037] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:37.597576 854 raft_consensus.cc:3037] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:37.598212 998 raft_consensus.cc:2454] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 1.
I20260808 02:22:37.598270 854 raft_consensus.cc:2454] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45a7827306e497b827b855eba7a5227 in term 1.
I20260808 02:22:37.598392 1070 leader_election.cc:302] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375ce3ceefd24c278478525e12b8fdb9, e45a7827306e497b827b855eba7a5227; no voters:
I20260808 02:22:37.598505 1269 raft_consensus.cc:2781] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:37.598575 1269 raft_consensus.cc:686] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 1 LEADER]: Becoming Leader. State: Replica: e45a7827306e497b827b855eba7a5227, State: Running, Role: LEADER
I20260808 02:22:37.598637 1269 consensus_queue.cc:237] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } }
I20260808 02:22:37.599224 692 catalog_manager.cc:5495] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 reported cstate change: term changed from 0 to 1, leader changed from <none> to e45a7827306e497b827b855eba7a5227 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "e45a7827306e497b827b855eba7a5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e45a7827306e497b827b855eba7a5227" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 38905 } health_report { overall_health: HEALTHY } } }
I20260808 02:22:37.687573 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:37.685361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.30.160.3:38905)
0808 02:22:37.685551 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:37.685552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:37.685559 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:37.685579 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:37.685579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:37.685589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:37.685647 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:37.685745 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:37.686208 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:37.686928 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:37.687104 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:37.687153 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:37.687287 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:37.687323 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:37.687362 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:37.687442 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 02:22:37.687460 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":125,"threads_started":1}
I20260808 02:22:37.689965 854 raft_consensus.cc:1261] T 25aa407ee07346dc87520ee5465ff408 P 18a884945bbe4cbaa058a1d3c5e27f3c [term 1 FOLLOWER]: Refusing update from remote peer e45a7827306e497b827b855eba7a5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:37.689965 998 raft_consensus.cc:1261] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Refusing update from remote peer e45a7827306e497b827b855eba7a5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:37.690194 1273 consensus_queue.cc:1035] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375ce3ceefd24c278478525e12b8fdb9" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:37.690320 1278 consensus_queue.cc:1035] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a884945bbe4cbaa058a1d3c5e27f3c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:22:37.694875 1270 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:22:39.213658 694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:39.213781 694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:39.214664 694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1d3db6e96a462ea43735b00441ff9d: Sending DeleteTablet for 3 replicas of tablet 25aa407ee07346dc87520ee5465ff408
I20260808 02:22:39.214965 1119 tablet_service.cc:1514] Processing DeleteTablet for tablet 25aa407ee07346dc87520ee5465ff408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:39 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:58190
I20260808 02:22:39.214967 833 tablet_service.cc:1514] Processing DeleteTablet for tablet 25aa407ee07346dc87520ee5465ff408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:39 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:49510
I20260808 02:22:39.215142 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 767
I20260808 02:22:39.215159 976 tablet_service.cc:1514] Processing DeleteTablet for tablet 25aa407ee07346dc87520ee5465ff408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:39 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:38574
I20260808 02:22:39.216152 1348 tablet_replica.cc:330] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: stopping tablet replica
I20260808 02:22:39.216266 1348 raft_consensus.cc:2229] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:39.216353 1348 raft_consensus.cc:2258] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:39.217087 1348 ts_tablet_manager.cc:1905] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:39.219364 1348 ts_tablet_manager.cc:1918] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:22:39.219449 1348 log.cc:1196] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/25aa407ee07346dc87520ee5465ff408
I20260808 02:22:39.219729 1348 ts_tablet_manager.cc:1939] T 25aa407ee07346dc87520ee5465ff408 P 375ce3ceefd24c278478525e12b8fdb9: Deleting consensus metadata
I20260808 02:22:39.220130 678 catalog_manager.cc:4841] TS 375ce3ceefd24c278478525e12b8fdb9 (127.30.160.2:45247): tablet 25aa407ee07346dc87520ee5465ff408 (table test-table [id=91935e5719504f248053cb1e3294e9e2]) successfully deleted
I20260808 02:22:39.221158 1346 tablet_replica.cc:330] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: stopping tablet replica
I20260808 02:22:39.221220 1346 raft_consensus.cc:2229] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:39.221292 1346 raft_consensus.cc:2258] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:39.221509 1346 ts_tablet_manager.cc:1905] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:39.222669 1346 ts_tablet_manager.cc:1918] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:22:39.222718 1346 log.cc:1196] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/25aa407ee07346dc87520ee5465ff408
I20260808 02:22:39.222954 1346 ts_tablet_manager.cc:1939] T 25aa407ee07346dc87520ee5465ff408 P e45a7827306e497b827b855eba7a5227: Deleting consensus metadata
I20260808 02:22:39.223713 681 catalog_manager.cc:4841] TS e45a7827306e497b827b855eba7a5227 (127.30.160.3:38905): tablet 25aa407ee07346dc87520ee5465ff408 (table test-table [id=91935e5719504f248053cb1e3294e9e2]) successfully deleted
W20260808 02:22:39.227527 680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:38087 (error 108)
I20260808 02:22:39.227788 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 911
I20260808 02:22:39.228538 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:39.227936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:38087 (local address 127.0.0.1:49528)
0808 02:22:39.228462 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:39.228504 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:38087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":61,"threads_started":1}
W20260808 02:22:39.228657 680 catalog_manager.cc:4584] TS 18a884945bbe4cbaa058a1d3c5e27f3c (127.30.160.1:38087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25aa407ee07346dc87520ee5465ff408: Network error: Client connection negotiation failed: client connection to 127.30.160.1:38087: connect: Connection refused (error 111)
I20260808 02:22:39.233937 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 1054
I20260808 02:22:39.240505 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 662
2026-08-08T02:22:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6090 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:39 dist-test-slave-zkpd krb5kdc[1353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:39 dist-test-slave-zkpd krb5kdc[1353](info): set up 2 sockets
Aug 08 02:22:39 dist-test-slave-zkpd krb5kdc[1353](info): commencing operation
krb5kdc: starting...
W20260808 02:22:40.307746 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:40.424889 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39931
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:40.506747 1366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:40.506953 1366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:40.506975 1366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:40.508131 1366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:40.508170 1366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:40.508185 1366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:40.508198 1366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:40.508210 1366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:40.509734 1366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:40.509977 1366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:40.510218 1366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:40.515661 1373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:40.515637 1375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:40.515682 1366 server_base.cc:1034] running on GCE node
W20260808 02:22:40.515687 1372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:40.516120 1366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:40.516460 1366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:40.517624 1366 hybrid_clock.cc:648] HybridClock initialized: now 1786155760517610 us; error 43 us; skew 500 ppm
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:40.525553 1366 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:40.526036 1366 webserver.cc:460] Webserver started at http://127.30.160.62:32789/ using document root <none> and password file <none>
I20260808 02:22:40.526275 1366 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:40.526345 1366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:40.526494 1366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:40.527550 1366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "23e1cd99e99d4c208c4a82139c8d54c3"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "08acf5ed7c0d072d09db3685b414580f"
server_key_iv: "e93498555e1bf1e2d62cab9853bb4065"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.527987 1366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "23e1cd99e99d4c208c4a82139c8d54c3"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "08acf5ed7c0d072d09db3685b414580f"
server_key_iv: "e93498555e1bf1e2d62cab9853bb4065"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.529346 1366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:40.530100 1382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.530273 1366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:40.530375 1366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "23e1cd99e99d4c208c4a82139c8d54c3"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "08acf5ed7c0d072d09db3685b414580f"
server_key_iv: "e93498555e1bf1e2d62cab9853bb4065"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.530459 1366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:40.541257 1366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:40.543519 1366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:40.543680 1366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:40.547557 1366 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39931
I20260808 02:22:40.547587 1436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39931 every 8 connection(s)
I20260808 02:22:40.547977 1366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:40.548802 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:40.549388 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 1366
I20260808 02:22:40.549464 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:40.549649 31360 external_mini_cluster.cc:1428] Setting key 2286dfc756272d0723f11caf9e3e7225
I20260808 02:22:40.551944 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3: Bootstrap starting.
I20260808 02:22:40.552624 1437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:40.552927 1437 log.cc:824] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:40.553643 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3: No bootstrap required, opened a new log
I20260808 02:22:40.555814 1437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39931 } }
I20260808 02:22:40.556013 1437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:40.556042 1437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e1cd99e99d4c208c4a82139c8d54c3, State: Initialized, Role: FOLLOWER
I20260808 02:22:40.556190 1437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39931 } }
I20260808 02:22:40.556265 1437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:40.556288 1437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:40.556361 1437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:40.556989 1437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39931 } }
I20260808 02:22:40.557075 1437 leader_election.cc:302] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e1cd99e99d4c208c4a82139c8d54c3; no voters:
I20260808 02:22:40.557269 1437 leader_election.cc:288] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:40.557300 1442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:40.557413 1442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [term 1 LEADER]: Becoming Leader. State: Replica: 23e1cd99e99d4c208c4a82139c8d54c3, State: Running, Role: LEADER
I20260808 02:22:40.557509 1442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39931 } }
I20260808 02:22:40.557538 1437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:40.558101 1443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e1cd99e99d4c208c4a82139c8d54c3. Latest consensus state: current_term: 1 leader_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39931 } } }
I20260808 02:22:40.558298 1443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:40.558239 1444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e1cd99e99d4c208c4a82139c8d54c3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39931 } } }
I20260808 02:22:40.558434 1444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:40.558555 1452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:40.559093 1452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:40.561053 1452 catalog_manager.cc:1347] Generated new cluster ID: 4ff79f2fc70f46d1a13e19e37191fd35
I20260808 02:22:40.561123 1452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:40.563222 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:40.550182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.30.160.62:39931)
0808 02:22:40.550391 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:40.550401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:40.550434 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:22:40.550972 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:40.550980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:40.551040 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:40.551196 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:40.551772 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.553176 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.554024 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.554620 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.554736 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:40.557161 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:40.557166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:40.557166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:40.557177 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:40.560231 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:40.560669 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:40.560673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:40.560673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:40.560781 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:40.562291 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:40.562293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:40.562293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:40.562582 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:40.562722 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:40.562899 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:22:40.563024 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260808 02:22:40.567520 1452 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:40.568329 1452 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:40.576377 1452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23e1cd99e99d4c208c4a82139c8d54c3: Generated new TSK 0
I20260808 02:22:40.576538 1452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:40.602669 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39931
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:40.684582 1464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:40.684818 1464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:40.684875 1464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:40.686014 1464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:40.686074 1464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:40.686126 1464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:40.687549 1464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:40.687809 1464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:40.688052 1464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:40.692936 1470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:40.692970 1469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:40.693048 1464 server_base.cc:1034] running on GCE node
W20260808 02:22:40.692970 1472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:40.693373 1464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:40.693627 1464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:40.694799 1464 hybrid_clock.cc:648] HybridClock initialized: now 1786155760694800 us; error 48 us; skew 500 ppm
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:40.697474 1464 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:40.697896 1464 webserver.cc:460] Webserver started at http://127.30.160.1:41131/ using document root <none> and password file <none>
I20260808 02:22:40.698151 1464 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:40.698220 1464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:40.698374 1464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:40.699381 1464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "3053941552414fda9108e67b03bcc6dc"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "88a4f03e943137c413be787731c15993"
server_key_iv: "c131ba9f076e560264c4926aee5b0cc0"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.699750 1464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "3053941552414fda9108e67b03bcc6dc"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "88a4f03e943137c413be787731c15993"
server_key_iv: "c131ba9f076e560264c4926aee5b0cc0"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.701108 1464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:40.701831 1479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.702011 1464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:40.702109 1464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "3053941552414fda9108e67b03bcc6dc"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "88a4f03e943137c413be787731c15993"
server_key_iv: "c131ba9f076e560264c4926aee5b0cc0"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.702196 1464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:40.714149 1464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:40.716414 1464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:40.716571 1464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:40.716794 1464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:40.717092 1464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:40.717146 1464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.717186 1464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:40.717214 1464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.723070 1464 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:35467
I20260808 02:22:40.723140 1595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:35467 every 8 connection(s)
I20260808 02:22:40.723449 1464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:40.727949 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 1464
I20260808 02:22:40.728056 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:40.728264 31360 external_mini_cluster.cc:1428] Setting key a28eda14be1b1dee3994525d1beb73b9
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:40.733783 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:40.724189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:37827 (local address 127.30.160.62:39931)
0808 02:22:40.724270 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:40.724273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:40.724488 (+ 215us) server_negotiation.cc:407] Connection header received
0808 02:22:40.724956 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:40.724957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:40.724967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:40.725021 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:40.725945 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.726552 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.727554 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.727770 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.727851 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:40.730409 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:40.730411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:40.730411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:40.730420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:40.731957 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:40.732466 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:40.732467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:40.732467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:40.732520 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:40.732803 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:40.732804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:40.732805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:40.732930 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:40.732983 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:40.733454 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 02:22:40.733531 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:22:40.733798 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:40.724156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39931 (local address 127.30.160.1:37827)
0808 02:22:40.724343 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:40.724378 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:40.724874 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:40.725070 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:40.725073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:40.725628 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:40.725820 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:40.725824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.726643 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.726644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:40.727453 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:40.727455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.727882 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.727883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:40.728069 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:40.728676 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:40.728686 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:40.730317 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:40.732057 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:40.732064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:40.732066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:40.732399 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:40.732584 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:40.732585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:40.732586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:40.732732 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:40.732998 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:40.733004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:40.733170 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 02:22:40.733307 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 02:22:40.733505 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260808 02:22:40.734454 1596 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39931
I20260808 02:22:40.734566 1596 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:40.734748 1596 heartbeater.cc:511] Master 127.30.160.62:39931 requested a full tablet report, sending...
I20260808 02:22:40.735376 1400 ts_manager.cc:194] Registered new tserver with Master: 3053941552414fda9108e67b03bcc6dc (127.30.160.1:35467)
I20260808 02:22:40.736362 1400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:37827
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:40.755457 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39931
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:40.839637 1601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:40.839854 1601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:40.839876 1601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:40.841118 1601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:40.841166 1601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:40.841192 1601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:40.842602 1601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:40.842787 1601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:40.842964 1601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:40.847874 1606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:40.847909 1607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:40.847869 1609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:40.848065 1601 server_base.cc:1034] running on GCE node
I20260808 02:22:40.848340 1601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:40.848529 1601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:40.849680 1601 hybrid_clock.cc:648] HybridClock initialized: now 1786155760849667 us; error 27 us; skew 500 ppm
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:40.852130 1601 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:40.852553 1601 webserver.cc:460] Webserver started at http://127.30.160.2:37609/ using document root <none> and password file <none>
I20260808 02:22:40.852779 1601 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:40.852857 1601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:40.852993 1601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:40.854012 1601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "235c8f9d482f48a1a32ae5459d68fea5"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "b56238a25f5d2c7e0b2e8457de5c5464"
server_key_iv: "e2e370a3dd76400018bd295ee8b09ee0"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.854385 1601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "235c8f9d482f48a1a32ae5459d68fea5"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "b56238a25f5d2c7e0b2e8457de5c5464"
server_key_iv: "e2e370a3dd76400018bd295ee8b09ee0"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.855707 1601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:40.856513 1616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.856683 1601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.856783 1601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "235c8f9d482f48a1a32ae5459d68fea5"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "b56238a25f5d2c7e0b2e8457de5c5464"
server_key_iv: "e2e370a3dd76400018bd295ee8b09ee0"
server_key_version: "encryptionkey@0"
I20260808 02:22:40.856876 1601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:40.871665 1601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:40.873490 1601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:40.873653 1601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:40.873914 1601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:40.874212 1601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:40.874266 1601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.874307 1601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:40.874338 1601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:40.879844 1601 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:37735
I20260808 02:22:40.879940 1733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:37735 every 8 connection(s)
I20260808 02:22:40.880245 1601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:40.880666 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 1601
I20260808 02:22:40.880770 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:40.880991 31360 external_mini_cluster.cc:1428] Setting key 9f481288757706542104ae7df4767e4e
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
I20260808 02:22:40.890694 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:40.880993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:35839 (local address 127.30.160.62:39931)
0808 02:22:40.881073 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:40.881075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:40.881364 (+ 289us) server_negotiation.cc:407] Connection header received
0808 02:22:40.881999 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:40.882000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:40.882009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:40.882046 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:40.883323 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.884115 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.885075 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.885303 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.885363 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:40.887800 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:40.887801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:40.887802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:40.887809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:40.889270 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:40.889674 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:40.889676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:40.889677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:40.889739 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:40.890018 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:40.890019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:40.890019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:40.890134 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:40.890213 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:40.890500 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 02:22:40.890586 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:22:40.890843 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:40.880963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39931 (local address 127.30.160.2:35839)
0808 02:22:40.881172 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:40.881269 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:40.881886 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:40.882111 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:40.882115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:40.882948 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:40.883224 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:40.883231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.884207 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.884208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:40.884991 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:40.884993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:40.885401 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:40.885402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:40.885550 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:40.886083 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:40.886093 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:40.887700 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:40.889358 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:40.889366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:40.889369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:40.889596 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:40.889832 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:40.889833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:40.889834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:40.889948 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:40.890204 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:40.890211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:40.890356 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 02:22:40.890460 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:22:40.890614 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260808 02:22:40.891305 1734 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39931
I20260808 02:22:40.891393 1734 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:40.891592 1734 heartbeater.cc:511] Master 127.30.160.62:39931 requested a full tablet report, sending...
I20260808 02:22:40.892037 1400 ts_manager.cc:194] Registered new tserver with Master: 235c8f9d482f48a1a32ae5459d68fea5 (127.30.160.2:37735)
I20260808 02:22:40.892376 1400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:35839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:40.906605 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39931
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:40.988076 1740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:40.988265 1740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:40.988286 1740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:40.989347 1740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:40.989380 1740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:40.989405 1740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:40.990847 1740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:40.991053 1740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:40.991216 1740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:40.996156 1748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:40.996186 1745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:40.996186 1746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:40.996220 1740 server_base.cc:1034] running on GCE node
I20260808 02:22:40.996529 1740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:40.996728 1740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:40.997867 1740 hybrid_clock.cc:648] HybridClock initialized: now 1786155760997861 us; error 43 us; skew 500 ppm
Aug 08 02:22:40 dist-test-slave-zkpd krb5kdc[1353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:41.000466 1740 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:41.000990 1740 webserver.cc:460] Webserver started at http://127.30.160.3:45369/ using document root <none> and password file <none>
I20260808 02:22:41.001231 1740 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:41.001278 1740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:41.001405 1740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:41.002437 1740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "beee6ff31a5141d4ae4f605e99c5ec9f"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "2639f78e1d02da57b18ad73c0275342c"
server_key_iv: "95820c09709d9b9520fecac47fe167a7"
server_key_version: "encryptionkey@0"
I20260808 02:22:41.002796 1740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "beee6ff31a5141d4ae4f605e99c5ec9f"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "2639f78e1d02da57b18ad73c0275342c"
server_key_iv: "95820c09709d9b9520fecac47fe167a7"
server_key_version: "encryptionkey@0"
I20260808 02:22:41.004141 1740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:41.004882 1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:41.005088 1740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:41.005189 1740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "beee6ff31a5141d4ae4f605e99c5ec9f"
format_stamp: "Formatted at 2026-08-08 02:22:40 on dist-test-slave-zkpd"
server_key: "2639f78e1d02da57b18ad73c0275342c"
server_key_iv: "95820c09709d9b9520fecac47fe167a7"
server_key_version: "encryptionkey@0"
I20260808 02:22:41.005275 1740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:41.012770 1740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:41.014962 1740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:41.015121 1740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:41.015348 1740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:41.015640 1740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:41.015693 1740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:41.015733 1740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:41.015763 1740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:41.021711 1740 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:33617
I20260808 02:22:41.021780 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:33617 every 8 connection(s)
I20260808 02:22:41.022053 1740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:22:41 dist-test-slave-zkpd krb5kdc[1353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155760, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:41.031066 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:41.022737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:35769 (local address 127.30.160.62:39931)
0808 02:22:41.022800 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:41.022802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:41.023005 (+ 203us) server_negotiation.cc:407] Connection header received
0808 02:22:41.023511 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:41.023513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:41.023525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:41.023589 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:41.024668 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:41.025195 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:41.026181 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:41.026402 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:41.026480 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:41.028589 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:41.028591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:41.028591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:41.028598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:41.029843 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:41.030292 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:41.030293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:41.030293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:41.030346 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:41.030558 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:41.030559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:41.030559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:41.030659 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:41.030709 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:41.030940 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 02:22:41.031002 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:22:41.031262 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:41.022703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39931 (local address 127.30.160.3:35769)
0808 02:22:41.022891 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:41.022924 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:41.023407 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:41.023662 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:41.023665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:41.024388 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:41.024567 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:41.024572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:41.025292 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:41.025293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:41.026083 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:41.026086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:41.026488 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:41.026490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:41.026613 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:41.027052 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:41.027060 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:41.028485 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:41.029941 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:41.029949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:41.029952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:41.030222 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:41.030407 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:41.030407 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:41.030408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:41.030503 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:41.030715 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:41.030719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:41.030832 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 02:22:41.030912 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:22:41.031029 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260808 02:22:41.031433 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 1740
I20260808 02:22:41.031548 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:41.031741 1871 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39931
I20260808 02:22:41.031790 31360 external_mini_cluster.cc:1428] Setting key 0c13dda43728f07d9ba0fd16285f1e06
I20260808 02:22:41.031875 1871 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:41.032100 1871 heartbeater.cc:511] Master 127.30.160.62:39931 requested a full tablet report, sending...
I20260808 02:22:41.032567 1399 ts_manager.cc:194] Registered new tserver with Master: beee6ff31a5141d4ae4f605e99c5ec9f (127.30.160.3:33617)
I20260808 02:22:41.032956 1399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:35769
I20260808 02:22:41.043601 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:22:41.055601 1440 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:41.054611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.30.160.62:39931)
0808 02:22:41.054698 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:41.054701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:41.054795 (+ 94us) server_negotiation.cc:407] Connection header received
0808 02:22:41.054900 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:41.054902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:41.054913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:41.054996 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:41.055531 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53512: BlockingRecv error: recv got EOF from 127.0.0.1:53512 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:22:41.056955 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 1464
I20260808 02:22:41.062887 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 1601
I20260808 02:22:41.068148 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 1740
I20260808 02:22:41.073851 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 1366
2026-08-08T02:22:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1831 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:41 dist-test-slave-zkpd krb5kdc[1885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:41 dist-test-slave-zkpd krb5kdc[1885](info): set up 2 sockets
Aug 08 02:22:41 dist-test-slave-zkpd krb5kdc[1885](info): commencing operation
krb5kdc: starting...
W20260808 02:22:42.131354 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:42 dist-test-slave-zkpd krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:42.248494 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:41725
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:42.329176 1898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:42.329380 1898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:42.329403 1898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:42.330454 1898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:42.330487 1898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:42.330502 1898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:42.330513 1898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:42.330525 1898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:42.331911 1898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:42.332094 1898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:42.332275 1898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:42.336957 1904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:42.336941 1903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:42.336933 1906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:42.337262 1898 server_base.cc:1034] running on GCE node
I20260808 02:22:42.337496 1898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:42.337723 1898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:42.338865 1898 hybrid_clock.cc:648] HybridClock initialized: now 1786155762338852 us; error 29 us; skew 500 ppm
Aug 08 02:22:42 dist-test-slave-zkpd krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155762, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:42.341393 1898 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:42.341871 1898 webserver.cc:460] Webserver started at http://127.30.160.62:40439/ using document root <none> and password file <none>
I20260808 02:22:42.342047 1898 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:42.342087 1898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:42.342175 1898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:42.343132 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f"
format_stamp: "Formatted at 2026-08-08 02:22:42 on dist-test-slave-zkpd"
server_key: "e533cec801e697207d1840be54eee8f7"
server_key_iv: "6d863caace630dbe34671ac91c4c2e46"
server_key_version: "encryptionkey@0"
I20260808 02:22:42.343429 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f"
format_stamp: "Formatted at 2026-08-08 02:22:42 on dist-test-slave-zkpd"
server_key: "e533cec801e697207d1840be54eee8f7"
server_key_iv: "6d863caace630dbe34671ac91c4c2e46"
server_key_version: "encryptionkey@0"
I20260808 02:22:42.344758 1898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:42.345443 1913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:42.345613 1898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:42.345717 1898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f"
format_stamp: "Formatted at 2026-08-08 02:22:42 on dist-test-slave-zkpd"
server_key: "e533cec801e697207d1840be54eee8f7"
server_key_iv: "6d863caace630dbe34671ac91c4c2e46"
server_key_version: "encryptionkey@0"
I20260808 02:22:42.345793 1898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:42.365463 1898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:42.368104 1898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:42.368284 1898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:42.372668 1898 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:41725
I20260808 02:22:42.372716 1966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:41725 every 8 connection(s)
I20260808 02:22:42.373230 1898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:42.373905 1967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:42.376181 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f: Bootstrap starting.
I20260808 02:22:42.376765 1967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:42.377065 1967 log.cc:824] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:42.377735 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f: No bootstrap required, opened a new log
I20260808 02:22:42.379298 1967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41725 } }
I20260808 02:22:42.379446 1967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:42.379467 1967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2dc1fd3cb747baae93e9d9cf9e8a5f, State: Initialized, Role: FOLLOWER
I20260808 02:22:42.379637 1967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41725 } }
I20260808 02:22:42.379699 1967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:42.379721 1967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:42.379786 1967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:42.380666 1967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41725 } }
I20260808 02:22:42.380776 1967 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2dc1fd3cb747baae93e9d9cf9e8a5f; no voters:
I20260808 02:22:42.380959 1967 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:42.381047 1970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:42.381210 1970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [term 1 LEADER]: Becoming Leader. State: Replica: 3b2dc1fd3cb747baae93e9d9cf9e8a5f, State: Running, Role: LEADER
I20260808 02:22:42.381244 1967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:42.381335 1970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41725 } }
I20260808 02:22:42.381747 1972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2dc1fd3cb747baae93e9d9cf9e8a5f. Latest consensus state: current_term: 1 leader_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41725 } } }
I20260808 02:22:42.381835 1972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:42.381995 1971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2dc1fd3cb747baae93e9d9cf9e8a5f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41725 } } }
I20260808 02:22:42.382058 1971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:42.382330 1979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:42.382794 1979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:42.383219 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 1898
I20260808 02:22:42.383328 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:42.383512 31360 external_mini_cluster.cc:1428] Setting key cf19e4e22bccbd0a57326a947ec4c2dd
I20260808 02:22:42.384426 1979 catalog_manager.cc:1347] Generated new cluster ID: cbed38a3bf7943f984ef4bb48580b9fb
I20260808 02:22:42.384479 1979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:42 dist-test-slave-zkpd krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:42.399142 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:42.383910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.30.160.62:41725)
0808 02:22:42.387964 (+ 4054us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:42.387973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:42.388132 (+ 159us) server_negotiation.cc:407] Connection header received
0808 02:22:42.388400 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:42.388407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:42.388493 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:42.388631 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:42.389558 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:42.391007 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:42.392054 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:42.392714 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:42.392834 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:42.395545 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:42.395550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:42.395550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:42.395562 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:42.397441 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:42.397849 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:42.397852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:42.397852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:42.397963 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:42.398176 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:42.398177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:42.398177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:42.398443 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:42.398582 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:42.398702 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 02:22:42.398845 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3966,"thread_start_us":80,"threads_started":1}
I20260808 02:22:42.400046 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:42.406430 1979 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:42.407152 1979 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:42.416916 1979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2dc1fd3cb747baae93e9d9cf9e8a5f: Generated new TSK 0
I20260808 02:22:42.417119 1979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:42.417908 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:42.413782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.30.160.62:41725)
0808 02:22:42.413823 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:42.413825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:42.413964 (+ 139us) server_negotiation.cc:407] Connection header received
0808 02:22:42.413998 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:42.413999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:42.414012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:42.414077 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:42.414464 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:42.415222 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:42.416295 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:42.416641 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:42.416711 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:42.417417 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:42.417418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:42.417419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:42.417627 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:42.417643 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:42.417651 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:42.417747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:42.417803 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 02:22:42.417829 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:22:42.420199 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 1898
2026-08-08T02:22:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1344 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:42 dist-test-slave-zkpd krb5kdc[2000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:42 dist-test-slave-zkpd krb5kdc[2000](info): set up 2 sockets
Aug 08 02:22:42 dist-test-slave-zkpd krb5kdc[2000](info): commencing operation
krb5kdc: starting...
W20260808 02:22:43.486197 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:43 dist-test-slave-zkpd krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:43.606410 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:33309
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:33309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:43.688407 2015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:43.688645 2015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:43.688683 2015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:43.689790 2015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:43.689860 2015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:43.689883 2015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:43.689904 2015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:43.689944 2015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:43.691375 2015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:33309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:33309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:43.691648 2015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:43.691879 2015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:43.696758 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:43.696772 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:43.696748 2023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:43.696936 2015 server_base.cc:1034] running on GCE node
I20260808 02:22:43.697247 2015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:43.697582 2015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:43.698729 2015 hybrid_clock.cc:648] HybridClock initialized: now 1786155763698711 us; error 31 us; skew 500 ppm
Aug 08 02:22:43 dist-test-slave-zkpd krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:43.705541 2015 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:43.706020 2015 webserver.cc:460] Webserver started at http://127.30.160.62:38671/ using document root <none> and password file <none>
I20260808 02:22:43.706264 2015 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:43.706341 2015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:43.706494 2015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:43.707536 2015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "7c1c7db217af4a3c882eef29504df530"
format_stamp: "Formatted at 2026-08-08 02:22:43 on dist-test-slave-zkpd"
server_key: "e30e42bd6e804fc29ef3946df0d32303"
server_key_iv: "31e37d9a3a9301d857fc0aee14d019eb"
server_key_version: "encryptionkey@0"
I20260808 02:22:43.707953 2015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "7c1c7db217af4a3c882eef29504df530"
format_stamp: "Formatted at 2026-08-08 02:22:43 on dist-test-slave-zkpd"
server_key: "e30e42bd6e804fc29ef3946df0d32303"
server_key_iv: "31e37d9a3a9301d857fc0aee14d019eb"
server_key_version: "encryptionkey@0"
I20260808 02:22:43.709332 2015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:43.710065 2030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:43.710263 2015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:43.710384 2015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "7c1c7db217af4a3c882eef29504df530"
format_stamp: "Formatted at 2026-08-08 02:22:43 on dist-test-slave-zkpd"
server_key: "e30e42bd6e804fc29ef3946df0d32303"
server_key_iv: "31e37d9a3a9301d857fc0aee14d019eb"
server_key_version: "encryptionkey@0"
I20260808 02:22:43.710466 2015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:43.720924 2015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:43.722873 2015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:43.723035 2015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:43.727105 2015 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:33309
I20260808 02:22:43.727137 2083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:33309 every 8 connection(s)
I20260808 02:22:43.727516 2015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:43.728492 2084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:43.730702 2084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530: Bootstrap starting.
I20260808 02:22:43.731194 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2015
I20260808 02:22:43.731314 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:43.731402 2084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:43.731519 31360 external_mini_cluster.cc:1428] Setting key c924689744aa65e8b4d9be47daf90929
I20260808 02:22:43.731740 2084 log.cc:824] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:43.732504 2084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530: No bootstrap required, opened a new log
I20260808 02:22:43.734390 2084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1c7db217af4a3c882eef29504df530" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33309 } }
I20260808 02:22:43.734524 2084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:43.734545 2084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c7db217af4a3c882eef29504df530, State: Initialized, Role: FOLLOWER
I20260808 02:22:43.734705 2084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1c7db217af4a3c882eef29504df530" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33309 } }
I20260808 02:22:43.734815 2084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:43.734856 2084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:43.734907 2084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:43.735447 2084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1c7db217af4a3c882eef29504df530" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33309 } }
I20260808 02:22:43.735569 2084 leader_election.cc:302] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c1c7db217af4a3c882eef29504df530; no voters:
I20260808 02:22:43.735752 2084 leader_election.cc:288] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:43.735788 2089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:43.735993 2089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [term 1 LEADER]: Becoming Leader. State: Replica: 7c1c7db217af4a3c882eef29504df530, State: Running, Role: LEADER
I20260808 02:22:43.736037 2084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:43.736410 2089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1c7db217af4a3c882eef29504df530" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33309 } }
I20260808 02:22:43.736835 2091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c1c7db217af4a3c882eef29504df530. Latest consensus state: current_term: 1 leader_uuid: "7c1c7db217af4a3c882eef29504df530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1c7db217af4a3c882eef29504df530" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33309 } } }
I20260808 02:22:43.737502 2091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:43.737164 2090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c1c7db217af4a3c882eef29504df530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1c7db217af4a3c882eef29504df530" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33309 } } }
I20260808 02:22:43.737723 2090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:43.737927 2106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:43.738519 2106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 02:22:43 dist-test-slave-zkpd krb5kdc[2000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:43.739989 2106 catalog_manager.cc:1347] Generated new cluster ID: a3c92b3b37ef4ccd9f9992ccbf0e65fc
I20260808 02:22:43.740043 2106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:43.743273 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:43.732080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.30.160.62:33309)
0808 02:22:43.732265 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:43.732274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:43.732306 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:22:43.732797 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:43.732811 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:43.732870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:43.733017 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:43.733650 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:43.734547 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:43.735391 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:43.736020 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:43.736126 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:43.739134 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:43.739138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:43.739138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:43.739150 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:43.741250 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:43.741665 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:43.741668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:43.741668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:43.741769 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:43.741998 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:43.741999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:43.741999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:43.742305 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:43.742464 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:43.742920 (+ 456us) server_negotiation.cc:299] Negotiation successful
0808 02:22:43.743052 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260808 02:22:43.762611 2106 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:43.763365 2106 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:43.769695 2106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c1c7db217af4a3c882eef29504df530: Generated new TSK 0
I20260808 02:22:43.769883 2106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:43.804927 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:33309
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:43.891814 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:43.892086 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:43.892110 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:43.893275 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:43.893317 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:43.893344 2112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:43.894819 2112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:43.895081 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:43.895326 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:43.900250 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:43.900261 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:43.900331 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:43.900640 2112 server_base.cc:1034] running on GCE node
I20260808 02:22:43.900836 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:43.901038 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:43.902176 2112 hybrid_clock.cc:648] HybridClock initialized: now 1786155763902159 us; error 33 us; skew 500 ppm
Aug 08 02:22:43 dist-test-slave-zkpd krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:43.904924 2112 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:43.905355 2112 webserver.cc:460] Webserver started at http://127.30.160.1:38657/ using document root <none> and password file <none>
I20260808 02:22:43.905599 2112 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:43.905669 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:43.905824 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:43.906889 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "fe2e4d20237c4bd5bc2966c10abfbdb8"
format_stamp: "Formatted at 2026-08-08 02:22:43 on dist-test-slave-zkpd"
server_key: "065de4a24dd4debf2703136b68dbf838"
server_key_iv: "fa7a85d0f83bdc8727160e0bbbba6750"
server_key_version: "encryptionkey@0"
I20260808 02:22:43.907258 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "fe2e4d20237c4bd5bc2966c10abfbdb8"
format_stamp: "Formatted at 2026-08-08 02:22:43 on dist-test-slave-zkpd"
server_key: "065de4a24dd4debf2703136b68dbf838"
server_key_iv: "fa7a85d0f83bdc8727160e0bbbba6750"
server_key_version: "encryptionkey@0"
I20260808 02:22:43.908656 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:43.909366 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:43.909541 2112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:43.909642 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "fe2e4d20237c4bd5bc2966c10abfbdb8"
format_stamp: "Formatted at 2026-08-08 02:22:43 on dist-test-slave-zkpd"
server_key: "065de4a24dd4debf2703136b68dbf838"
server_key_iv: "fa7a85d0f83bdc8727160e0bbbba6750"
server_key_version: "encryptionkey@0"
I20260808 02:22:43.909721 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:43.923135 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:43.925415 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:43.925573 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:43.925791 2112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:43.926095 2112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:43.926149 2112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:43.926180 2112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:43.926209 2112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:43.932017 2112 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:38815
I20260808 02:22:43.932090 2243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:38815 every 8 connection(s)
I20260808 02:22:43.932406 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:22:43 dist-test-slave-zkpd krb5kdc[2000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155763, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:43.940448 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2112
I20260808 02:22:43.940558 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:43.940745 31360 external_mini_cluster.cc:1428] Setting key 2c77ce8867fef4950d29394142f1d212
I20260808 02:22:43.941955 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:43.933162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:33309 (local address 127.30.160.1:58463)
0808 02:22:43.933348 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:43.933392 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:43.933899 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:43.934147 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:43.934153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:43.934700 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:43.934885 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:43.934890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:43.935609 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:43.935611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:43.936543 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:43.936545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:43.937045 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:43.937046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:43.937174 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:43.937560 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:43.937568 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:43.938947 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:43.940590 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:43.940598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:43.940600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:43.940880 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:43.941103 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:43.941104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:43.941104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:43.941203 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:43.941461 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:43.941467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:43.941598 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 02:22:43.941684 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:22:43.941797 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260808 02:22:43.941972 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:43.933202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:58463 (local address 127.30.160.62:33309)
0808 02:22:43.933285 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:43.933288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:43.933472 (+ 184us) server_negotiation.cc:407] Connection header received
0808 02:22:43.934004 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:43.934005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:43.934016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:43.934069 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:43.934987 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:43.935510 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:43.936635 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:43.936950 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:43.937023 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:43.939029 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:43.939030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:43.939031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:43.939037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:43.940482 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:43.940963 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:43.940964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:43.940964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:43.941019 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:43.941266 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:43.941267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:43.941267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:43.941386 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:43.941469 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:43.941732 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:22:43.941837 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:22:43.942752 2244 heartbeater.cc:347] Connected to a master server at 127.30.160.62:33309
I20260808 02:22:43.942859 2244 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:43.943051 2244 heartbeater.cc:511] Master 127.30.160.62:33309 requested a full tablet report, sending...
I20260808 02:22:43.943634 2047 ts_manager.cc:194] Registered new tserver with Master: fe2e4d20237c4bd5bc2966c10abfbdb8 (127.30.160.1:38815)
I20260808 02:22:43.944607 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:58463
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:43.966694 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:33309
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:44.049001 2249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:44.049229 2249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:44.049266 2249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:44.050396 2249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:44.050462 2249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:44.050508 2249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:44.052059 2249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:44.052330 2249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:44.052558 2249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:44.057417 2258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:44.057462 2255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:44.057492 2256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:44.057824 2249 server_base.cc:1034] running on GCE node
I20260808 02:22:44.058032 2249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:44.058266 2249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:44.059438 2249 hybrid_clock.cc:648] HybridClock initialized: now 1786155764059421 us; error 30 us; skew 500 ppm
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:44.062011 2249 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:44.062454 2249 webserver.cc:460] Webserver started at http://127.30.160.2:44533/ using document root <none> and password file <none>
I20260808 02:22:44.062659 2249 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:44.062698 2249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:44.062793 2249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:44.063731 2249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "341cd97310b2476a9cde39fceba7a198"
format_stamp: "Formatted at 2026-08-08 02:22:44 on dist-test-slave-zkpd"
server_key: "4d43d41ac0e04bd6f52a12b0a30e0e67"
server_key_iv: "813a55bf55d242de7975835a7dfcc455"
server_key_version: "encryptionkey@0"
I20260808 02:22:44.064107 2249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "341cd97310b2476a9cde39fceba7a198"
format_stamp: "Formatted at 2026-08-08 02:22:44 on dist-test-slave-zkpd"
server_key: "4d43d41ac0e04bd6f52a12b0a30e0e67"
server_key_iv: "813a55bf55d242de7975835a7dfcc455"
server_key_version: "encryptionkey@0"
I20260808 02:22:44.065404 2249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:44.066112 2265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:44.066299 2249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:44.066385 2249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "341cd97310b2476a9cde39fceba7a198"
format_stamp: "Formatted at 2026-08-08 02:22:44 on dist-test-slave-zkpd"
server_key: "4d43d41ac0e04bd6f52a12b0a30e0e67"
server_key_iv: "813a55bf55d242de7975835a7dfcc455"
server_key_version: "encryptionkey@0"
I20260808 02:22:44.066439 2249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:44.076031 2249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:44.078215 2249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:44.078333 2249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:44.078507 2249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:44.078744 2249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:44.078774 2249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:44.078794 2249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:44.078806 2249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:44.084841 2249 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:32933
I20260808 02:22:44.084920 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:32933 every 8 connection(s)
I20260808 02:22:44.085229 2249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:44.091852 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2249
I20260808 02:22:44.091991 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:44.092206 31360 external_mini_cluster.cc:1428] Setting key 6769fe30eaca61fcdf00389a8924244d
I20260808 02:22:44.094483 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:44.085927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:49287 (local address 127.30.160.62:33309)
0808 02:22:44.085979 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:44.085981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:44.086195 (+ 214us) server_negotiation.cc:407] Connection header received
0808 02:22:44.086649 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:44.086650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:44.086660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:44.086713 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:44.087581 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.088137 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.089111 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.089302 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.089366 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:44.091418 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:44.091420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:44.091420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:44.091426 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:44.093026 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:44.093516 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:44.093517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:44.093517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:44.093584 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:44.093885 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:44.093885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:44.093886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:44.093985 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:44.094036 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:44.094355 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:22:44.094419 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:22:44.094792 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:44.085909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:33309 (local address 127.30.160.2:49287)
0808 02:22:44.086070 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:44.086104 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:44.086566 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:44.086764 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:44.086767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:44.087320 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:44.087505 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:44.087509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.088214 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.088214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:44.089010 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:44.089012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.089382 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.089383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:44.089476 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:44.089953 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:44.089962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:44.091332 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:44.093101 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:44.093109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:44.093111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:44.093431 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:44.093655 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:44.093656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:44.093656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:44.093815 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:44.094048 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:44.094055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:44.094216 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 02:22:44.094336 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:22:44.094512 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260808 02:22:44.095497 2380 heartbeater.cc:347] Connected to a master server at 127.30.160.62:33309
I20260808 02:22:44.095575 2380 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:44.095769 2380 heartbeater.cc:511] Master 127.30.160.62:33309 requested a full tablet report, sending...
I20260808 02:22:44.096302 2047 ts_manager.cc:194] Registered new tserver with Master: 341cd97310b2476a9cde39fceba7a198 (127.30.160.2:32933)
I20260808 02:22:44.096802 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:49287
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:44.118616 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:33309
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:44.200045 2385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:44.200243 2385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:44.200268 2385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:44.201412 2385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:44.201454 2385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:44.201508 2385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:44.202915 2385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:44.203153 2385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:44.203389 2385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:44.208407 2390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:44.208330 2391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:44.208345 2394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:44.208557 2385 server_base.cc:1034] running on GCE node
I20260808 02:22:44.208741 2385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:44.208981 2385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:44.210143 2385 hybrid_clock.cc:648] HybridClock initialized: now 1786155764210120 us; error 34 us; skew 500 ppm
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:44.212668 2385 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:44.213199 2385 webserver.cc:460] Webserver started at http://127.30.160.3:39523/ using document root <none> and password file <none>
I20260808 02:22:44.213428 2385 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:44.213496 2385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:44.213627 2385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:44.214771 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "7734422c035b491e8ea4451a9fd006c4"
format_stamp: "Formatted at 2026-08-08 02:22:44 on dist-test-slave-zkpd"
server_key: "3021c5a95cf566045082560f7703d687"
server_key_iv: "2579ff1cc86cd1ca750cf5a75a8a5afb"
server_key_version: "encryptionkey@0"
I20260808 02:22:44.215160 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "7734422c035b491e8ea4451a9fd006c4"
format_stamp: "Formatted at 2026-08-08 02:22:44 on dist-test-slave-zkpd"
server_key: "3021c5a95cf566045082560f7703d687"
server_key_iv: "2579ff1cc86cd1ca750cf5a75a8a5afb"
server_key_version: "encryptionkey@0"
I20260808 02:22:44.216583 2385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:44.217425 2401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:44.217643 2385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:44.217748 2385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "7734422c035b491e8ea4451a9fd006c4"
format_stamp: "Formatted at 2026-08-08 02:22:44 on dist-test-slave-zkpd"
server_key: "3021c5a95cf566045082560f7703d687"
server_key_iv: "2579ff1cc86cd1ca750cf5a75a8a5afb"
server_key_version: "encryptionkey@0"
I20260808 02:22:44.217842 2385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:44.229748 2385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:44.231485 2385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:44.231645 2385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:44.231870 2385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:44.232290 2385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:44.232349 2385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:44.232383 2385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:44.232410 2385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:44.238899 2385 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:46783
I20260808 02:22:44.238938 2516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:46783 every 8 connection(s)
I20260808 02:22:44.239264 2385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:44.243250 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2385
I20260808 02:22:44.243366 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:44.243564 31360 external_mini_cluster.cc:1428] Setting key 1a0bef8376df4c2e7aa87c255d29fcad
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:44.248661 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:44.239987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:59983 (local address 127.30.160.62:33309)
0808 02:22:44.240054 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:44.240056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:44.240266 (+ 210us) server_negotiation.cc:407] Connection header received
0808 02:22:44.240819 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:44.240821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:44.240835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:44.240886 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:44.241847 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.242356 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.243449 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.243703 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.243775 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:44.246154 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:44.246155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:44.246155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:44.246162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:44.247430 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:44.247808 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:44.247809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:44.247809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:44.247867 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:44.248136 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:44.248137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:44.248137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:44.248238 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:44.248290 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:44.248520 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 02:22:44.248597 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:22:44.248908 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:44.239977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:33309 (local address 127.30.160.3:59983)
0808 02:22:44.240149 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:44.240184 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:44.240707 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:44.240968 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:44.240972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:44.241544 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:44.241728 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:44.241733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.242453 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.242454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:44.243371 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:44.243375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.243812 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.243814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:44.244024 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:44.244637 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:44.244647 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:44.246057 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:44.247508 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:44.247515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:44.247517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:44.247737 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:44.247967 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:44.247967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:44.247968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:44.248078 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:44.248297 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:44.248301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:44.248412 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 02:22:44.248492 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:22:44.248644 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260808 02:22:44.249337 2517 heartbeater.cc:347] Connected to a master server at 127.30.160.62:33309
I20260808 02:22:44.249446 2517 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:44.249625 2517 heartbeater.cc:511] Master 127.30.160.62:33309 requested a full tablet report, sending...
I20260808 02:22:44.249974 2047 ts_manager.cc:194] Registered new tserver with Master: 7734422c035b491e8ea4451a9fd006c4 (127.30.160.3:46783)
I20260808 02:22:44.250337 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:59983
I20260808 02:22:44.255221 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155764, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155764, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:44.281200 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:44.271985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.30.160.62:33309)
0808 02:22:44.272053 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:44.272055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:44.272191 (+ 136us) server_negotiation.cc:407] Connection header received
0808 02:22:44.272282 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:44.272283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:44.272295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:44.272376 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:44.272957 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.273449 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.274613 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:44.274968 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:44.275047 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:44.277854 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:44.277856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:44.277856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:44.277865 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:44.279756 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:44.280245 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:44.280246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:44.280246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:44.280323 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:44.280578 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:44.280579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:44.280579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:44.280833 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:44.280932 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:44.281071 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 02:22:44.281135 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 02:22:44.281414 2047 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47428
I20260808 02:22:44.281811 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2112
I20260808 02:22:44.292064 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2249
I20260808 02:22:44.298005 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2385
I20260808 02:22:44.303402 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2015
2026-08-08T02:22:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1884 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2529](info): set up 2 sockets
Aug 08 02:22:44 dist-test-slave-zkpd krb5kdc[2529](info): commencing operation
krb5kdc: starting...
W20260808 02:22:45.367534 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:45.492503 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45101
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:45.573783 2544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:45.573978 2544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:45.573999 2544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:45.575157 2544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:45.575198 2544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:45.575212 2544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:45.575224 2544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:45.575235 2544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:45.576773 2544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:45.577031 2544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:45.577246 2544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:45.582231 2550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:45.582204 2552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:45.582216 2549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:45.582249 2544 server_base.cc:1034] running on GCE node
I20260808 02:22:45.582696 2544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:45.582989 2544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:45.584128 2544 hybrid_clock.cc:648] HybridClock initialized: now 1786155765584108 us; error 36 us; skew 500 ppm
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:45.591159 2544 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:45.591605 2544 webserver.cc:460] Webserver started at http://127.30.160.62:44221/ using document root <none> and password file <none>
I20260808 02:22:45.591856 2544 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:45.591969 2544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:45.592132 2544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:45.593212 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "dd408687ec0c32210a6b0de69d9703cc"
server_key_iv: "0c05f7424c33c08f84367a5a8b76ee27"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.593575 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "dd408687ec0c32210a6b0de69d9703cc"
server_key_iv: "0c05f7424c33c08f84367a5a8b76ee27"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.594911 2544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:45.595642 2559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.595819 2544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:45.595939 2544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "dd408687ec0c32210a6b0de69d9703cc"
server_key_iv: "0c05f7424c33c08f84367a5a8b76ee27"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.596024 2544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:45.614176 2544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:45.616046 2544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:45.616212 2544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:45.620342 2544 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:45101
I20260808 02:22:45.620383 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:45101 every 8 connection(s)
I20260808 02:22:45.620911 2544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:45.621601 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:45.624212 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2: Bootstrap starting.
I20260808 02:22:45.624848 2613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:45.625123 2613 log.cc:824] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:45.625855 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2: No bootstrap required, opened a new log
I20260808 02:22:45.627369 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2544
I20260808 02:22:45.627511 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:45.627731 31360 external_mini_cluster.cc:1428] Setting key f76aacadc626180b204127ccb7bd29e6
I20260808 02:22:45.627858 2613 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45101 } }
I20260808 02:22:45.628098 2613 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:45.628149 2613 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77b8e57f58f40eebe4afd0c2adfc3e2, State: Initialized, Role: FOLLOWER
I20260808 02:22:45.628314 2613 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45101 } }
I20260808 02:22:45.628425 2613 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:45.628464 2613 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:45.628528 2613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:45.629257 2613 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45101 } }
I20260808 02:22:45.629393 2613 leader_election.cc:302] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77b8e57f58f40eebe4afd0c2adfc3e2; no voters:
I20260808 02:22:45.629606 2613 leader_election.cc:288] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:45.629642 2618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:45.629765 2618 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [term 1 LEADER]: Becoming Leader. State: Replica: c77b8e57f58f40eebe4afd0c2adfc3e2, State: Running, Role: LEADER
I20260808 02:22:45.629858 2618 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45101 } }
I20260808 02:22:45.629938 2613 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:45.630198 2620 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45101 } } }
I20260808 02:22:45.630468 2620 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:45.630394 2619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77b8e57f58f40eebe4afd0c2adfc3e2. Latest consensus state: current_term: 1 leader_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77b8e57f58f40eebe4afd0c2adfc3e2" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45101 } } }
I20260808 02:22:45.630723 2619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:45.631101 2626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:45.631731 2626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:45.633572 2626 catalog_manager.cc:1347] Generated new cluster ID: 73d1759ea7be47acae40e255de8ff931
I20260808 02:22:45.633631 2626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:45.640252 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:45.628265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.30.160.62:45101)
0808 02:22:45.628948 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:45.628956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:45.628982 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:22:45.629072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:45.629078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:45.629116 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:45.629256 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:45.630077 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.631554 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.632636 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.633296 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.633448 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:45.636147 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:45.636152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:45.636152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:45.636165 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:45.638507 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:45.638942 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:45.638945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:45.638946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:45.639052 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:45.639307 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:45.639308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:45.639309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:45.639561 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:45.639766 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:45.639877 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:22:45.640065 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":591,"threads_started":1}
I20260808 02:22:45.651491 2626 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:45.652091 2626 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:45.657456 2626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77b8e57f58f40eebe4afd0c2adfc3e2: Generated new TSK 0
I20260808 02:22:45.657631 2626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:45.687747 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45101
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:45.769047 2639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:45.769241 2639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:45.769265 2639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:45.770332 2639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:45.770364 2639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:45.770390 2639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:45.771867 2639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:45.772056 2639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:45.772224 2639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:45.776909 2647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:45.776909 2644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:45.776930 2645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:45.777093 2639 server_base.cc:1034] running on GCE node
I20260808 02:22:45.777364 2639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:45.777602 2639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:45.778767 2639 hybrid_clock.cc:648] HybridClock initialized: now 1786155765778754 us; error 28 us; skew 500 ppm
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:45.781271 2639 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:45.781715 2639 webserver.cc:460] Webserver started at http://127.30.160.1:44603/ using document root <none> and password file <none>
I20260808 02:22:45.781941 2639 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:45.781986 2639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:45.782073 2639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:45.783008 2639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "9dd42ef39b4142b988df3926e2a5c004"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "7c1f5eac043e0732e04addba07c13e3c"
server_key_iv: "264444924d1968ccef1480aaa039215b"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.783296 2639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "9dd42ef39b4142b988df3926e2a5c004"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "7c1f5eac043e0732e04addba07c13e3c"
server_key_iv: "264444924d1968ccef1480aaa039215b"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.784612 2639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 02:22:45.785287 2654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.785480 2639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:45.785563 2639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "9dd42ef39b4142b988df3926e2a5c004"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "7c1f5eac043e0732e04addba07c13e3c"
server_key_iv: "264444924d1968ccef1480aaa039215b"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.785617 2639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:45.798300 2639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:45.800633 2639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:45.800740 2639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:45.800915 2639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:45.801213 2639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:45.801247 2639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.801290 2639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:45.801316 2639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.806731 2639 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:45343
I20260808 02:22:45.806840 2770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:45343 every 8 connection(s)
I20260808 02:22:45.807127 2639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:45.812270 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2639
I20260808 02:22:45.812405 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:45.812625 31360 external_mini_cluster.cc:1428] Setting key 563574862e142d18ca60f7902deb1416
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:45.816627 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:45.807871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:39805 (local address 127.30.160.62:45101)
0808 02:22:45.807934 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:45.807937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:45.808139 (+ 202us) server_negotiation.cc:407] Connection header received
0808 02:22:45.809010 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:45.809011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:45.809024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:45.809072 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:45.809615 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.810105 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.811012 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.811211 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.811262 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:45.813486 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:45.813488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:45.813488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:45.813499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:45.815122 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:45.815632 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:45.815633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:45.815633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:45.815686 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:45.816000 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:45.816000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:45.816000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:45.816100 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:45.816154 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:45.816481 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 02:22:45.816550 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:22:45.816829 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:45.807858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45101 (local address 127.30.160.1:39805)
0808 02:22:45.808022 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:45.808057 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:45.808925 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:45.809129 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:45.809132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:45.809368 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:45.809546 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:45.809550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.810180 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.810181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:45.810941 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:45.810943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.811281 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.811281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:45.811386 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:45.811789 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:45.811796 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:45.813383 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:45.815222 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:45.815230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:45.815232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:45.815551 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:45.815754 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:45.815755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:45.815755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:45.815931 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:45.816167 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:45.816173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:45.816337 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 02:22:45.816447 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:22:45.816609 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260808 02:22:45.817366 2771 heartbeater.cc:347] Connected to a master server at 127.30.160.62:45101
I20260808 02:22:45.817445 2771 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:45.817638 2771 heartbeater.cc:511] Master 127.30.160.62:45101 requested a full tablet report, sending...
I20260808 02:22:45.818172 2577 ts_manager.cc:194] Registered new tserver with Master: 9dd42ef39b4142b988df3926e2a5c004 (127.30.160.1:45343)
I20260808 02:22:45.819185 2577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:39805
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:45.838483 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45101
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:45.920145 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:45.920346 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:45.920369 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:45.921444 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:45.921475 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:45.921500 2776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:45.922921 2776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:45.923101 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:45.923271 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:45.928187 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:45.928202 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:45.928277 2776 server_base.cc:1034] running on GCE node
W20260808 02:22:45.928243 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:45.928617 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:45.928846 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:45.929962 2776 hybrid_clock.cc:648] HybridClock initialized: now 1786155765929950 us; error 27 us; skew 500 ppm
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:45.932523 2776 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:45.932963 2776 webserver.cc:460] Webserver started at http://127.30.160.2:34079/ using document root <none> and password file <none>
I20260808 02:22:45.933178 2776 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:45.933221 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:45.933310 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:45.934269 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "bb2eae4e67dd434b9599f97affdecc3d"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "478a2e7051f5226ad56d37ea9e324809"
server_key_iv: "582cf3f0686e776dece9715501456609"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.934574 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "bb2eae4e67dd434b9599f97affdecc3d"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "478a2e7051f5226ad56d37ea9e324809"
server_key_iv: "582cf3f0686e776dece9715501456609"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.935803 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:45.936542 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.936712 2776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.936782 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "bb2eae4e67dd434b9599f97affdecc3d"
format_stamp: "Formatted at 2026-08-08 02:22:45 on dist-test-slave-zkpd"
server_key: "478a2e7051f5226ad56d37ea9e324809"
server_key_iv: "582cf3f0686e776dece9715501456609"
server_key_version: "encryptionkey@0"
I20260808 02:22:45.936833 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:45.957645 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:45.960031 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:45.960153 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:45.960335 2776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:45.960711 2776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:45.960748 2776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.960793 2776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:45.960822 2776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:45.966514 2776 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:45355
I20260808 02:22:45.966600 2907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:45355 every 8 connection(s)
I20260808 02:22:45.966904 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:22:45 dist-test-slave-zkpd krb5kdc[2529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155765, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:45.973659 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2776
I20260808 02:22:45.973791 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:45.974002 31360 external_mini_cluster.cc:1428] Setting key 6da0045a7bdf0840ff471dc0b4186223
I20260808 02:22:45.976234 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:45.967711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:59645 (local address 127.30.160.62:45101)
0808 02:22:45.967792 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:45.967794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:45.967983 (+ 189us) server_negotiation.cc:407] Connection header received
0808 02:22:45.968930 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:45.968931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:45.968941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:45.968993 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:45.969551 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.970040 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.970968 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.971146 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.971195 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:45.973248 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:45.973249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:45.973250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:45.973256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:45.974817 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:45.975293 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:45.975293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:45.975294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:45.975354 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:45.975626 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:45.975627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:45.975627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:45.975722 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:45.975772 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:45.976102 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 02:22:45.976169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:22:45.976538 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:45.967664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45101 (local address 127.30.160.2:59645)
0808 02:22:45.967846 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:45.967881 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:45.968839 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:45.969047 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:45.969050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:45.969287 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:45.969480 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:45.969484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.970116 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.970116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:45.970896 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:45.970897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:45.971215 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:45.971215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:45.971336 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:45.971750 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:45.971758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:45.973164 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:45.974899 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:45.974907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:45.974909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:45.975215 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:45.975418 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:45.975419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:45.975420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:45.975563 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:45.975785 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:45.975791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:45.975966 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 02:22:45.976080 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:22:45.976253 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260808 02:22:45.976965 2908 heartbeater.cc:347] Connected to a master server at 127.30.160.62:45101
I20260808 02:22:45.977053 2908 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:45.977247 2908 heartbeater.cc:511] Master 127.30.160.62:45101 requested a full tablet report, sending...
I20260808 02:22:45.977628 2577 ts_manager.cc:194] Registered new tserver with Master: bb2eae4e67dd434b9599f97affdecc3d (127.30.160.2:45355)
I20260808 02:22:45.978005 2577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:59645
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:45.998436 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45101
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:46.080663 2913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:46.080861 2913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:46.080884 2913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:46.082082 2913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:46.082125 2913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:46.082182 2913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:46.083693 2913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:46.083971 2913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:46.084203 2913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:46.089108 2918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:46.089114 2919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:46.089114 2921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:46.089325 2913 server_base.cc:1034] running on GCE node
I20260808 02:22:46.089524 2913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:46.089715 2913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:46.090852 2913 hybrid_clock.cc:648] HybridClock initialized: now 1786155766090847 us; error 45 us; skew 500 ppm
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[2529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155766, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:46.093415 2913 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:46.093801 2913 webserver.cc:460] Webserver started at http://127.30.160.3:42037/ using document root <none> and password file <none>
I20260808 02:22:46.094003 2913 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:46.094043 2913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:46.094129 2913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:46.095072 2913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "599854bdd8df4ee9812063adf8903252"
format_stamp: "Formatted at 2026-08-08 02:22:46 on dist-test-slave-zkpd"
server_key: "2b0f22388aec7144a689c59f5f7c89ff"
server_key_iv: "60668bf0a59d59d6251f9db2578486ec"
server_key_version: "encryptionkey@0"
I20260808 02:22:46.095376 2913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "599854bdd8df4ee9812063adf8903252"
format_stamp: "Formatted at 2026-08-08 02:22:46 on dist-test-slave-zkpd"
server_key: "2b0f22388aec7144a689c59f5f7c89ff"
server_key_iv: "60668bf0a59d59d6251f9db2578486ec"
server_key_version: "encryptionkey@0"
I20260808 02:22:46.096688 2913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:46.097355 2928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:46.097527 2913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:46.097631 2913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "599854bdd8df4ee9812063adf8903252"
format_stamp: "Formatted at 2026-08-08 02:22:46 on dist-test-slave-zkpd"
server_key: "2b0f22388aec7144a689c59f5f7c89ff"
server_key_iv: "60668bf0a59d59d6251f9db2578486ec"
server_key_version: "encryptionkey@0"
I20260808 02:22:46.097704 2913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:46.107360 2913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:46.109210 2913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:46.109375 2913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:46.109592 2913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:46.109894 2913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:46.109946 2913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:46.109987 2913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:46.110015 2913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:46.116035 2913 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:35467
I20260808 02:22:46.116103 3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:35467 every 8 connection(s)
I20260808 02:22:46.116441 2913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[2529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155766, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:46.123119 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 2913
I20260808 02:22:46.123204 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:46.123407 31360 external_mini_cluster.cc:1428] Setting key 01250812a0c65b6e8ca3efb57556a3d5
I20260808 02:22:46.125898 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:46.117222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:59253 (local address 127.30.160.62:45101)
0808 02:22:46.117267 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:46.117269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:46.117474 (+ 205us) server_negotiation.cc:407] Connection header received
0808 02:22:46.118436 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:46.118438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:46.118451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:46.118506 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:46.119123 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.119630 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.120701 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.120903 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.120979 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:46.123003 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:46.123004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:46.123004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:46.123011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:46.124637 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:46.125028 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:46.125029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:46.125030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:46.125085 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:46.125328 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:46.125329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:46.125329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:46.125448 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:46.125514 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:46.125766 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 02:22:46.125834 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:46.126020 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:46.117203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45101 (local address 127.30.160.3:59253)
0808 02:22:46.117358 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:46.117394 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:46.118325 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:46.118580 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:46.118583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:46.118834 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:46.119017 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:46.119021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.119725 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.119726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:46.120591 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:46.120594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.121000 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.121001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:46.121120 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:46.121550 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:46.121558 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:46.122888 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:46.124717 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:46.124723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:46.124726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:46.124953 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:46.125149 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:46.125150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:46.125150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:46.125245 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:46.125515 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:46.125519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:46.125644 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:22:46.125723 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:22:46.125841 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260808 02:22:46.126482 3045 heartbeater.cc:347] Connected to a master server at 127.30.160.62:45101
I20260808 02:22:46.126569 3045 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:46.126766 3045 heartbeater.cc:511] Master 127.30.160.62:45101 requested a full tablet report, sending...
I20260808 02:22:46.127153 2577 ts_manager.cc:194] Registered new tserver with Master: 599854bdd8df4ee9812063adf8903252 (127.30.160.3:35467)
I20260808 02:22:46.127532 2577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:59253
I20260808 02:22:46.135175 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[2529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[2529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:46.159066 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:46.151252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.30.160.1:45343)
0808 02:22:46.151448 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:46.151453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:46.151462 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:46.151485 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:46.151486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:46.151499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:46.151582 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:46.152253 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.152991 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.153775 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.154103 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.154198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:46.156342 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:46.156350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:46.156350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:46.156357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:46.157986 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:46.158314 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:46.158317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:46.158318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:46.158388 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:46.158557 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:46.158558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:46.158558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:46.158681 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:46.158782 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:46.158870 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 02:22:46.158932 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[2529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:46.167092 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:46.160481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.30.160.62:45101)
0808 02:22:46.160540 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:46.160541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:46.160550 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:46.160581 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:46.160582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:46.160590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:46.160636 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:46.161198 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.161671 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.162309 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:46.162490 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:46.162539 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:46.164702 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:46.164704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:46.164704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:46.164710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:46.166059 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:46.166370 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:46.166371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:46.166371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:46.166422 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:46.166627 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:46.166627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:46.166627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:46.166759 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:46.166852 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:46.166922 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:22:46.166984 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 02:22:46.167336 2577 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60226
I20260808 02:22:46.167625 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2639
I20260808 02:22:46.173909 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2776
I20260808 02:22:46.178829 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2913
I20260808 02:22:46.183732 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 2544
2026-08-08T02:22:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1880 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[3057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[3057](info): set up 2 sockets
Aug 08 02:22:46 dist-test-slave-zkpd krb5kdc[3057](info): commencing operation
krb5kdc: starting...
W20260808 02:22:47.231709 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:47.350382 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40337
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:40337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:47.431993 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:47.432282 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:47.432307 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:47.433470 3070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:47.433509 3070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:47.433523 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:47.433537 3070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:47.433591 3070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:47.435045 3070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:40337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:47.435298 3070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:47.435521 3070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:47.440286 3076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.440291 3078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.440310 3075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:47.440475 3070 server_base.cc:1034] running on GCE node
I20260808 02:22:47.440690 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:47.440968 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:47.442126 3070 hybrid_clock.cc:648] HybridClock initialized: now 1786155767442108 us; error 37 us; skew 500 ppm
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:47.444568 3070 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:47.444940 3070 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:47.445010 3070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:47.445163 3070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:47.446178 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "64a04d40bc724cd0988f31451f244388"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "7dc29f64c3961afe64259af0e7915da2"
server_key_iv: "9920f0c8fdf610c16de9c8021dda69ba"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.446558 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "64a04d40bc724cd0988f31451f244388"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "7dc29f64c3961afe64259af0e7915da2"
server_key_iv: "9920f0c8fdf610c16de9c8021dda69ba"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.447840 3070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:22:47.448577 3084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.448745 3070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:47.448848 3070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "64a04d40bc724cd0988f31451f244388"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "7dc29f64c3961afe64259af0e7915da2"
server_key_iv: "9920f0c8fdf610c16de9c8021dda69ba"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.448928 3070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:47.463173 3070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:47.465854 3070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:47.466019 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:47.470232 3070 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:40337
I20260808 02:22:47.470256 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:40337 every 8 connection(s)
I20260808 02:22:47.470561 3070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:47.471607 3137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:47.474011 3137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388: Bootstrap starting.
I20260808 02:22:47.474614 3137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:47.474949 3137 log.cc:824] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:47.475116 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3070
I20260808 02:22:47.475245 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:47.475467 31360 external_mini_cluster.cc:1428] Setting key 57e8b54ee9bc30d44e0fb0dacdbb7788
I20260808 02:22:47.475822 3137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388: No bootstrap required, opened a new log
I20260808 02:22:47.477563 3137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a04d40bc724cd0988f31451f244388" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40337 } }
I20260808 02:22:47.477725 3137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:47.477766 3137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a04d40bc724cd0988f31451f244388, State: Initialized, Role: FOLLOWER
I20260808 02:22:47.477937 3137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a04d40bc724cd0988f31451f244388" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40337 } }
I20260808 02:22:47.478032 3137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:47.478070 3137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:47.478122 3137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:47.479048 3137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a04d40bc724cd0988f31451f244388" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40337 } }
I20260808 02:22:47.479141 3137 leader_election.cc:302] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a04d40bc724cd0988f31451f244388; no voters:
I20260808 02:22:47.479344 3137 leader_election.cc:288] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:47.479372 3142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:47.479482 3142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [term 1 LEADER]: Becoming Leader. State: Replica: 64a04d40bc724cd0988f31451f244388, State: Running, Role: LEADER
I20260808 02:22:47.479584 3142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a04d40bc724cd0988f31451f244388" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40337 } }
I20260808 02:22:47.479636 3137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:47.479972 3142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a04d40bc724cd0988f31451f244388. Latest consensus state: current_term: 1 leader_uuid: "64a04d40bc724cd0988f31451f244388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a04d40bc724cd0988f31451f244388" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40337 } } }
I20260808 02:22:47.479981 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a04d40bc724cd0988f31451f244388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a04d40bc724cd0988f31451f244388" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40337 } } }
I20260808 02:22:47.480064 3142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:47.480088 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:47.480309 3148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:47.480904 3148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:47.482502 3148 catalog_manager.cc:1347] Generated new cluster ID: 764f61ca580948508ffac0fe4239f6b8
I20260808 02:22:47.482565 3148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:47.488163 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.475994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.30.160.62:40337)
0808 02:22:47.476200 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:47.476210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:47.476240 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:22:47.477546 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:47.477553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:47.477600 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:47.477714 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:47.478531 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.479655 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.480786 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.481387 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.481470 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.484251 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:47.484255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:47.484255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:47.484266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:47.486256 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.486663 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.486666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.486667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.486797 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.487024 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.487025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.487025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.487342 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:47.487514 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:47.487592 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:22:47.487730 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260808 02:22:47.514796 3148 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:47.515344 3148 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:47.521445 3148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388: Generated new TSK 0
I20260808 02:22:47.521639 3148 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:47.547307 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40337
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:47.632601 3163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:47.632825 3163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:47.632861 3163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:47.633991 3163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:47.634055 3163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:47.634102 3163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:47.635572 3163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:47.635819 3163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:47.636055 3163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:47.640978 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.641016 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.640987 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:47.641119 3163 server_base.cc:1034] running on GCE node
I20260808 02:22:47.641379 3163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:47.641563 3163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:47.642731 3163 hybrid_clock.cc:648] HybridClock initialized: now 1786155767642717 us; error 28 us; skew 500 ppm
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:47.645339 3163 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:47.645646 3163 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:47.645715 3163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:47.645867 3163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:47.646883 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "ae2663b559bc4969b87302ca565615ca"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "45047dc4e57a9b1973bf64a7ee30dec7"
server_key_iv: "4ded9d1eb00efbe74d4999173d1e1384"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.647240 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "ae2663b559bc4969b87302ca565615ca"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "45047dc4e57a9b1973bf64a7ee30dec7"
server_key_iv: "4ded9d1eb00efbe74d4999173d1e1384"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.648608 3163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:47.649322 3178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.649495 3163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:47.649605 3163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "ae2663b559bc4969b87302ca565615ca"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "45047dc4e57a9b1973bf64a7ee30dec7"
server_key_iv: "4ded9d1eb00efbe74d4999173d1e1384"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.649677 3163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:47.657919 3163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:47.659606 3163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:47.659760 3163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:47.660038 3163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:47.660339 3163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:47.660392 3163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.660434 3163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:47.660461 3163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.666255 3163 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:33929
I20260808 02:22:47.666344 3292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:33929 every 8 connection(s)
I20260808 02:22:47.666579 3163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:47.672868 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3163
I20260808 02:22:47.673002 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:47.673187 31360 external_mini_cluster.cc:1428] Setting key 6f2e57eecf50b13359954e8dc41af4ed
I20260808 02:22:47.675926 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.667289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:47511 (local address 127.30.160.62:40337)
0808 02:22:47.667352 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:47.667354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:47.667522 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:22:47.668071 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:47.668071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:47.668084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:47.668139 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:47.668991 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.669468 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.670409 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.670606 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.670655 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.672683 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:47.672685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:47.672685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:47.672693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:47.674392 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.674878 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.674878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.674879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.674928 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.675204 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.675205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.675205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.675300 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:47.675350 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:47.675717 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 02:22:47.675816 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:22:47.676365 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.667260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40337 (local address 127.30.160.1:47511)
0808 02:22:47.667407 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:47.667445 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:47.667971 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:47.668189 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:47.668192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:47.668434 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:47.668903 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:47.668908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.669546 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.669547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:47.670311 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:47.670338 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.670679 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.670679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:47.670772 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.671197 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:47.671205 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:47.672595 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:47.674475 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:47.674483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:47.674485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:47.674801 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:47.674993 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:47.674994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:47.674994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:47.675133 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:47.675364 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:47.675370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:47.675527 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 02:22:47.676062 (+ 535us) client_negotiation.cc:240] Negotiation successful
0808 02:22:47.676213 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260808 02:22:47.676909 3293 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40337
I20260808 02:22:47.677007 3293 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:47.677160 3293 heartbeater.cc:511] Master 127.30.160.62:40337 requested a full tablet report, sending...
I20260808 02:22:47.677628 3101 ts_manager.cc:194] Registered new tserver with Master: ae2663b559bc4969b87302ca565615ca (127.30.160.1:33929)
I20260808 02:22:47.678232 3101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:47511
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:47.699172 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40337
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:47.781235 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:47.781431 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:47.781455 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:47.782601 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:47.782641 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:47.782670 3298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:47.784273 3298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:47.784516 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:47.784736 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:47.789594 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.789604 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.789598 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:47.789956 3298 server_base.cc:1034] running on GCE node
I20260808 02:22:47.790171 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:47.790365 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:47.791505 3298 hybrid_clock.cc:648] HybridClock initialized: now 1786155767791488 us; error 31 us; skew 500 ppm
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:47.794047 3298 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:47.794363 3298 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:47.794409 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:47.794572 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:47.795557 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "0bb092c7718f40958f9946915bf70159"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "519d5a24579fa0c65fb3ec78cdf68d4c"
server_key_iv: "f96d5fe19fc8087ca7890f0377ff0799"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.795966 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "0bb092c7718f40958f9946915bf70159"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "519d5a24579fa0c65fb3ec78cdf68d4c"
server_key_iv: "f96d5fe19fc8087ca7890f0377ff0799"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.797214 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:47.797919 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.798110 3298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:47.798199 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "0bb092c7718f40958f9946915bf70159"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "519d5a24579fa0c65fb3ec78cdf68d4c"
server_key_iv: "f96d5fe19fc8087ca7890f0377ff0799"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.798271 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:47.822180 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:47.824563 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:47.824731 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:47.824983 3298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:47.825299 3298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:47.825335 3298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.825378 3298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:47.825407 3298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.831068 3298 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:33483
I20260808 02:22:47.831122 3426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:33483 every 8 connection(s)
I20260808 02:22:47.831372 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:47.834211 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3298
I20260808 02:22:47.834339 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:47.834537 31360 external_mini_cluster.cc:1428] Setting key 7bb7700e7db58aec7599c652e7dca766
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:47.841765 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.832149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:37265 (local address 127.30.160.62:40337)
0808 02:22:47.832211 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:47.832213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:47.832435 (+ 222us) server_negotiation.cc:407] Connection header received
0808 02:22:47.832975 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:47.832976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:47.832989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:47.833049 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:47.834014 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.834814 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.835997 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.836303 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.836382 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.838732 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:47.838734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:47.838734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:47.838743 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:47.840276 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.840775 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.840776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.840776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.840839 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.841139 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.841140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.841140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.841237 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:47.841284 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:47.841637 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 02:22:47.841703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:22:47.842090 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.832117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40337 (local address 127.30.160.2:37265)
0808 02:22:47.832302 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:47.832346 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:47.832865 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:47.833116 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:47.833121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:47.833392 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:47.833907 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:47.833912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.834923 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.834924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:47.835868 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:47.835871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.836403 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.836404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:47.836506 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.837032 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:47.837044 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:47.838635 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:47.840360 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:47.840369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:47.840371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:47.840696 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:47.840909 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:47.840911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:47.840911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:47.841063 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:47.841303 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:47.841310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:47.841484 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 02:22:47.841609 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 02:22:47.841793 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260808 02:22:47.842588 3427 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40337
I20260808 02:22:47.842696 3427 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:47.842869 3427 heartbeater.cc:511] Master 127.30.160.62:40337 requested a full tablet report, sending...
I20260808 02:22:47.843242 3101 ts_manager.cc:194] Registered new tserver with Master: 0bb092c7718f40958f9946915bf70159 (127.30.160.2:33483)
I20260808 02:22:47.843614 3101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:37265
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:47.862495 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40337
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:47.944116 3432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:47.944319 3432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:47.944343 3432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:47.945485 3432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:47.945528 3432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:47.945580 3432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:47.947142 3432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:47.947381 3432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:47.947611 3432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:47.952600 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.952617 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:47.952638 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:47.952850 3432 server_base.cc:1034] running on GCE node
I20260808 02:22:47.953076 3432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:47.953342 3432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:47.954516 3432 hybrid_clock.cc:648] HybridClock initialized: now 1786155767954496 us; error 34 us; skew 500 ppm
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:47.957016 3432 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:47.957415 3432 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:47.957487 3432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:47.957629 3432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:47.958649 3432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "b3e3473839a6dcb09ec457c403bed2a6"
server_key_iv: "2e15dc55701041bc095d8c27ba6517ca"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.959028 3432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "b3e3473839a6dcb09ec457c403bed2a6"
server_key_iv: "2e15dc55701041bc095d8c27ba6517ca"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.960417 3432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:47.961136 3446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.961301 3432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:47.961408 3432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701"
format_stamp: "Formatted at 2026-08-08 02:22:47 on dist-test-slave-zkpd"
server_key: "b3e3473839a6dcb09ec457c403bed2a6"
server_key_iv: "2e15dc55701041bc095d8c27ba6517ca"
server_key_version: "encryptionkey@0"
I20260808 02:22:47.961486 3432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:47.978027 3432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:47.980324 3432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:47.980492 3432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:47.980732 3432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:47.981089 3432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:47.981144 3432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.981179 3432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:47.981208 3432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:47.987102 3432 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:34157
I20260808 02:22:47.987138 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:34157 every 8 connection(s)
I20260808 02:22:47.987416 3432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:47.987682 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3432
I20260808 02:22:47.987834 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:47.988108 31360 external_mini_cluster.cc:1428] Setting key 99c96d12138cf69ab4ee7dee2994f88c
Aug 08 02:22:47 dist-test-slave-zkpd krb5kdc[3057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:47.996865 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.988327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:36285 (local address 127.30.160.62:40337)
0808 02:22:47.988374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:47.988376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:47.988538 (+ 162us) server_negotiation.cc:407] Connection header received
0808 02:22:47.989187 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:47.989188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:47.989199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:47.989257 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:47.990190 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.990688 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.991674 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.991875 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.992002 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.994071 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:47.994073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:47.994073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:47.994081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:47.995459 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.995875 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.995876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.995877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.995971 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:47.996219 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:47.996220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:47.996220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:47.996340 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:47.996413 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:47.996684 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:22:47.996765 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:22:47.997047 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:47.988245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40337 (local address 127.30.160.3:36285)
0808 02:22:47.988424 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:47.988471 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:47.989085 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:47.989331 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:47.989335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:47.989595 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:47.990085 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:47.990090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.990787 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.990788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:47.991584 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:47.991587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:47.992013 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:47.992014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:47.992147 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:47.992645 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:47.992653 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:47.993976 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:47.995563 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:47.995571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:47.995573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:47.995789 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:47.996043 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:47.996043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:47.996044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:47.996152 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:47.996416 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:47.996423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:47.996556 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 02:22:47.996660 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:22:47.996815 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260808 02:22:47.997624 3562 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40337
I20260808 02:22:47.997705 3562 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:47.997871 3562 heartbeater.cc:511] Master 127.30.160.62:40337 requested a full tablet report, sending...
I20260808 02:22:47.998260 3101 ts_manager.cc:194] Registered new tserver with Master: 14fb3bb6f8934ad8bdc3dc7cf9db6701 (127.30.160.3:34157)
I20260808 02:22:47.998652 3101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:36285
I20260808 02:22:47.999847 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:22:48.006863 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.001239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.30.160.62:40337)
0808 02:22:48.001288 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.001289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.001443 (+ 154us) server_negotiation.cc:407] Connection header received
0808 02:22:48.001497 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.001498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.001506 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.001544 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.002096 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.002544 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.003218 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.003391 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.003439 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.004525 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.004526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.004527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:48.004532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.005935 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:48.006230 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:48.006230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:48.006231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:48.006288 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:48.006451 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:48.006452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:48.006452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:48.006578 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.006652 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.006737 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.006791 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 02:22:48.008370 3101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:48.009924 3101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:48.016239 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.012236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.30.160.3:34157)
0808 02:22:48.012443 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.012447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.012458 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:48.012498 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.012500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.012520 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.012600 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:48.012892 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.013750 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.015068 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.015771 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.015859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.015945 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.016114 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.016154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":104,"threads_started":1}
I20260808 02:22:48.016228 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.012093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34157 (local address 127.0.0.1:44858)
0808 02:22:48.012276 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.012319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.012386 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.012630 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.012634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.012636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:48.012703 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.012704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.013858 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.013859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.014992 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.014995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.015862 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.015863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.015973 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.015990 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.016096 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.016154 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260808 02:22:48.016590 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.012104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:33929 (local address 127.0.0.1:53660)
0808 02:22:48.012495 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.012504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.012543 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.012852 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.012853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.012855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:48.012902 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.012904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.014296 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.014297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.015280 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.015282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.016194 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.016195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.016258 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.016262 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.016471 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.016517 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260808 02:22:48.016767 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.012320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33483 (local address 127.0.0.1:42310)
0808 02:22:48.012702 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.012713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.012754 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.013117 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.013117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.013120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:48.013159 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.013161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.014430 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.014431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.015544 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.015547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.016530 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.016531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.016587 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.016590 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.016649 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.016692 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":36,"threads_started":1}
I20260808 02:22:48.016949 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.012490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.30.160.2:33483)
0808 02:22:48.012875 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.012880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.012890 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:48.012933 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.012935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.012950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.013026 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:48.013246 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.014328 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.015634 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.016377 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.016447 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.016498 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.016733 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.016780 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":92,"threads_started":1}
I20260808 02:22:48.016952 3495 tablet_service.cc:1467] Processing CreateTablet for tablet 42ccbbfe81f846378ba50901d414a6e9 (DEFAULT_TABLE table=test-table [id=9ece998d2042407a87d272c30b978ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:48.017124 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.012242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.30.160.1:33929)
0808 02:22:48.012668 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.012672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.012682 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:48.012724 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.012727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.012742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.013013 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:48.013105 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.014187 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.015363 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.016039 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.016107 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.016157 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.016948 (+ 791us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.017016 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":92,"threads_started":1}
I20260808 02:22:48.017364 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ccbbfe81f846378ba50901d414a6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.017642 3361 tablet_service.cc:1467] Processing CreateTablet for tablet 42ccbbfe81f846378ba50901d414a6e9 (DEFAULT_TABLE table=test-table [id=9ece998d2042407a87d272c30b978ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:48.017993 3361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ccbbfe81f846378ba50901d414a6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.017956 3226 tablet_service.cc:1467] Processing CreateTablet for tablet 42ccbbfe81f846378ba50901d414a6e9 (DEFAULT_TABLE table=test-table [id=9ece998d2042407a87d272c30b978ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:48.018257 3226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ccbbfe81f846378ba50901d414a6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.021389 3578 tablet_bootstrap.cc:492] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: Bootstrap starting.
I20260808 02:22:48.021454 3576 tablet_bootstrap.cc:492] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca: Bootstrap starting.
I20260808 02:22:48.021996 3576 tablet_bootstrap.cc:654] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.022073 3578 tablet_bootstrap.cc:654] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.022517 3576 log.cc:824] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.022555 3578 log.cc:824] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.022917 3577 tablet_bootstrap.cc:492] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Bootstrap starting.
I20260808 02:22:48.023300 3578 tablet_bootstrap.cc:492] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: No bootstrap required, opened a new log
I20260808 02:22:48.023386 3578 ts_tablet_manager.cc:1397] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:48.023492 3577 tablet_bootstrap.cc:654] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.024076 3577 log.cc:824] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.024307 3576 tablet_bootstrap.cc:492] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca: No bootstrap required, opened a new log
I20260808 02:22:48.024389 3576 ts_tablet_manager.cc:1397] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:22:48.024994 3577 tablet_bootstrap.cc:492] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: No bootstrap required, opened a new log
I20260808 02:22:48.025059 3577 ts_tablet_manager.cc:1397] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:22:48.025511 3578 raft_consensus.cc:348] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.025663 3578 raft_consensus.cc:374] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.025696 3578 raft_consensus.cc:729] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb092c7718f40958f9946915bf70159, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.025825 3578 consensus_queue.cc:260] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.025831 3576 raft_consensus.cc:348] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.025976 3576 raft_consensus.cc:374] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.026029 3576 raft_consensus.cc:729] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2663b559bc4969b87302ca565615ca, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.026115 3578 ts_tablet_manager.cc:1428] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:22:48.026137 3427 heartbeater.cc:503] Master 127.30.160.62:40337 was elected leader, sending a full tablet report...
I20260808 02:22:48.026136 3576 consensus_queue.cc:260] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.026430 3576 ts_tablet_manager.cc:1428] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:48.026511 3293 heartbeater.cc:503] Master 127.30.160.62:40337 was elected leader, sending a full tablet report...
I20260808 02:22:48.027037 3582 raft_consensus.cc:482] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:48.027112 3582 raft_consensus.cc:504] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.027061 3577 raft_consensus.cc:348] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.027195 3577 raft_consensus.cc:374] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.027226 3577 raft_consensus.cc:729] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fb3bb6f8934ad8bdc3dc7cf9db6701, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.027338 3577 consensus_queue.cc:260] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.027408 3582 leader_election.cc:288] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14fb3bb6f8934ad8bdc3dc7cf9db6701 (127.30.160.3:34157), ae2663b559bc4969b87302ca565615ca (127.30.160.1:33929)
I20260808 02:22:48.027611 3577 ts_tablet_manager.cc:1428] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:22:48.027733 3562 heartbeater.cc:503] Master 127.30.160.62:40337 was elected leader, sending a full tablet report...
I20260808 02:22:48.031066 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.027634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:58129 (local address 127.30.160.1:33929)
0808 02:22:48.027670 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.027673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.027856 (+ 183us) server_negotiation.cc:407] Connection header received
0808 02:22:48.027975 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.027976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.027985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.028016 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:48.028200 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.028877 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.030179 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.030728 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.030808 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.030816 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.030974 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.031006 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:22:48.031066 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.027681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:48283 (local address 127.30.160.3:34157)
0808 02:22:48.027718 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.027720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.027729 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:48.027752 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.027752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.027762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.027816 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:48.028020 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.028780 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.030179 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.030729 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.030814 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.030822 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.030973 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.031007 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:22:48.031108 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.027519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34157 (local address 127.30.160.2:48283)
0808 02:22:48.027559 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.027570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.027609 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.027863 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.027864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.027865 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:48.027944 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.027946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.028869 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.028870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.030086 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.030089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.030817 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.030818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.030899 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.030904 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.030963 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.031024 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":31}
I20260808 02:22:48.031114 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.027595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:33929 (local address 127.30.160.2:58129)
0808 02:22:48.027792 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.027802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.027879 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.028074 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.028074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.028076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:48.028129 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.028131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.028960 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.028960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.030086 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.030089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.030828 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.030829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.030899 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.030903 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.030962 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.031022 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260808 02:22:48.031432 3515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ccbbfe81f846378ba50901d414a6e9" candidate_uuid: "0bb092c7718f40958f9946915bf70159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" is_pre_election: true
I20260808 02:22:48.031562 3515 raft_consensus.cc:2454] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb092c7718f40958f9946915bf70159 in term 0.
I20260808 02:22:48.031625 3246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ccbbfe81f846378ba50901d414a6e9" candidate_uuid: "0bb092c7718f40958f9946915bf70159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2663b559bc4969b87302ca565615ca" is_pre_election: true
I20260808 02:22:48.031742 3246 raft_consensus.cc:2454] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb092c7718f40958f9946915bf70159 in term 0.
I20260808 02:22:48.031821 3316 leader_election.cc:302] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb092c7718f40958f9946915bf70159, 14fb3bb6f8934ad8bdc3dc7cf9db6701; no voters:
I20260808 02:22:48.031963 3582 raft_consensus.cc:2781] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:48.032018 3582 raft_consensus.cc:482] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:48.032040 3582 raft_consensus.cc:3037] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.032580 3582 raft_consensus.cc:504] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.032717 3582 leader_election.cc:288] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [CANDIDATE]: Term 1 election: Requested vote from peers 14fb3bb6f8934ad8bdc3dc7cf9db6701 (127.30.160.3:34157), ae2663b559bc4969b87302ca565615ca (127.30.160.1:33929)
I20260808 02:22:48.032871 3515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ccbbfe81f846378ba50901d414a6e9" candidate_uuid: "0bb092c7718f40958f9946915bf70159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701"
I20260808 02:22:48.032893 3246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ccbbfe81f846378ba50901d414a6e9" candidate_uuid: "0bb092c7718f40958f9946915bf70159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2663b559bc4969b87302ca565615ca"
I20260808 02:22:48.032946 3515 raft_consensus.cc:3037] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.032954 3246 raft_consensus.cc:3037] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.033783 3515 raft_consensus.cc:2454] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb092c7718f40958f9946915bf70159 in term 1.
I20260808 02:22:48.033783 3246 raft_consensus.cc:2454] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb092c7718f40958f9946915bf70159 in term 1.
I20260808 02:22:48.033967 3316 leader_election.cc:302] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb092c7718f40958f9946915bf70159, 14fb3bb6f8934ad8bdc3dc7cf9db6701; no voters:
I20260808 02:22:48.034103 3582 raft_consensus.cc:2781] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:48.034231 3582 raft_consensus.cc:686] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb092c7718f40958f9946915bf70159, State: Running, Role: LEADER
I20260808 02:22:48.034329 3582 consensus_queue.cc:237] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } }
I20260808 02:22:48.035023 3101 catalog_manager.cc:5495] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb092c7718f40958f9946915bf70159 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "0bb092c7718f40958f9946915bf70159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bb092c7718f40958f9946915bf70159" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33483 } health_report { overall_health: HEALTHY } } }
I20260808 02:22:48.044407 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.041896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.30.160.2:33483)
0808 02:22:48.041959 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.041962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.041971 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:48.042044 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.042045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.042063 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.042149 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:48.042297 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.042762 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.043540 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.043746 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.043830 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.044042 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:48.044145 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:48.044234 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.044285 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.044305 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:22:48.046113 3515 raft_consensus.cc:1261] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 1 FOLLOWER]: Refusing update from remote peer 0bb092c7718f40958f9946915bf70159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:48.046139 3246 raft_consensus.cc:1261] T 42ccbbfe81f846378ba50901d414a6e9 P ae2663b559bc4969b87302ca565615ca [term 1 FOLLOWER]: Refusing update from remote peer 0bb092c7718f40958f9946915bf70159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:48.046478 3586 consensus_queue.cc:1035] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae2663b559bc4969b87302ca565615ca" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:48.046603 3582 consensus_queue.cc:1035] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14fb3bb6f8934ad8bdc3dc7cf9db6701" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:48.048895 3589 mvcc.cc:205] Tried to move back new op lower bound from 7316094025914114048 to 7316094025869144064. Current Snapshot: MvccSnapshot[applied={T|T < 7316094025914114048}]
I20260808 02:22:48.049489 3590 mvcc.cc:205] Tried to move back new op lower bound from 7316094025914114048 to 7316094025869144064. Current Snapshot: MvccSnapshot[applied={T|T < 7316094025914114048}]
I20260808 02:22:48.049388 3591 mvcc.cc:205] Tried to move back new op lower bound from 7316094025914114048 to 7316094025869144064. Current Snapshot: MvccSnapshot[applied={T|T < 7316094025914114048}]
I20260808 02:22:48.051138 3099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:48.051236 3099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:48.052110 3099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a04d40bc724cd0988f31451f244388: Sending DeleteTablet for 3 replicas of tablet 42ccbbfe81f846378ba50901d414a6e9
I20260808 02:22:48.052421 3495 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ccbbfe81f846378ba50901d414a6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:48 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:44858
I20260808 02:22:48.052469 3361 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ccbbfe81f846378ba50901d414a6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:48 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260808 02:22:48.052671 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3163
I20260808 02:22:48.052771 3226 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ccbbfe81f846378ba50901d414a6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:48 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:53660
I20260808 02:22:48.054149 3600 tablet_replica.cc:330] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: stopping tablet replica
I20260808 02:22:48.054289 3600 raft_consensus.cc:2229] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:48.054445 3600 raft_consensus.cc:2258] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:48.054967 3600 ts_tablet_manager.cc:1905] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:48.056896 3600 ts_tablet_manager.cc:1918] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:48.057030 3600 log.cc:1196] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/42ccbbfe81f846378ba50901d414a6e9
I20260808 02:22:48.057381 3600 ts_tablet_manager.cc:1939] T 42ccbbfe81f846378ba50901d414a6e9 P 14fb3bb6f8934ad8bdc3dc7cf9db6701: Deleting consensus metadata
I20260808 02:22:48.057921 3087 catalog_manager.cc:4841] TS 14fb3bb6f8934ad8bdc3dc7cf9db6701 (127.30.160.3:34157): tablet 42ccbbfe81f846378ba50901d414a6e9 (table test-table [id=9ece998d2042407a87d272c30b978ef8]) successfully deleted
W20260808 02:22:48.059706 3085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:33929 (error 108)
I20260808 02:22:48.060031 3601 tablet_replica.cc:330] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: stopping tablet replica
I20260808 02:22:48.060135 3601 raft_consensus.cc:2229] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:48.060213 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.060081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:33929 (local address 127.0.0.1:53676)
0808 02:22:48.060137 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.060182 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:33929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:22:48.060240 3601 raft_consensus.cc:2258] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:22:48.060343 3085 catalog_manager.cc:4584] TS ae2663b559bc4969b87302ca565615ca (127.30.160.1:33929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42ccbbfe81f846378ba50901d414a6e9: Network error: Client connection negotiation failed: client connection to 127.30.160.1:33929: connect: Connection refused (error 111)
I20260808 02:22:48.060503 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3298
I20260808 02:22:48.060763 3601 ts_tablet_manager.cc:1905] T 42ccbbfe81f846378ba50901d414a6e9 P 0bb092c7718f40958f9946915bf70159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:48.066571 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3432
I20260808 02:22:48.066665 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.066547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33483 (local address 127.0.0.1:42324)
0808 02:22:48.066603 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.066632 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.2:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 02:22:48.072059 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3070
2026-08-08T02:22:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1888 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:48Z Disabled control of system clock
I20260808 02:22:48.101161 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:41851
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:41851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:22:48.185328 3606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:48.185544 3606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:48.185576 3606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:48.186756 3606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:48.186794 3606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:48.186810 3606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:48.186823 3606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:48.186836 3606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:48.188287 3606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:41851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:41851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:48.188493 3606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:48.188755 3606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:48.193761 3614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.193810 3612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.193805 3611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:48.194142 3606 server_base.cc:1034] running on GCE node
I20260808 02:22:48.194432 3606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:48.194761 3606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:48.195964 3606 hybrid_clock.cc:648] HybridClock initialized: now 1786155768195945 us; error 47 us; skew 500 ppm
I20260808 02:22:48.197221 3606 webserver.cc:460] Webserver started at http://127.30.160.62:37599/ using document root <none> and password file <none>
I20260808 02:22:48.197494 3606 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:48.197566 3606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:48.197708 3606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:48.198756 3606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "2a0430dd477e446aa962d8ce4388ca60"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "d1f96aa59d42ba42d502b4f62bac5b41"
server_key_iv: "86a46e06fbdade8f4a92e28db8e9a3b3"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.199127 3606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "2a0430dd477e446aa962d8ce4388ca60"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "d1f96aa59d42ba42d502b4f62bac5b41"
server_key_iv: "86a46e06fbdade8f4a92e28db8e9a3b3"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.200525 3606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:48.201259 3620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.201433 3606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:48.201538 3606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "2a0430dd477e446aa962d8ce4388ca60"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "d1f96aa59d42ba42d502b4f62bac5b41"
server_key_iv: "86a46e06fbdade8f4a92e28db8e9a3b3"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.201624 3606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:48.204985 3606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:48.205271 3606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:48.205432 3606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:48.209373 3606 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:41851
I20260808 02:22:48.209404 3675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:41851 every 8 connection(s)
I20260808 02:22:48.209733 3606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:48.210359 3676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.213147 3676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60: Bootstrap starting.
I20260808 02:22:48.213734 3676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.214048 3676 log.cc:824] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.214715 3676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60: No bootstrap required, opened a new log
I20260808 02:22:48.215405 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3606
I20260808 02:22:48.215539 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:48.215734 31360 external_mini_cluster.cc:1428] Setting key fbd3408fb7689068ff289edc0186716b
I20260808 02:22:48.216784 3676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0430dd477e446aa962d8ce4388ca60" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41851 } }
I20260808 02:22:48.216974 3676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.217017 3676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0430dd477e446aa962d8ce4388ca60, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.217156 3676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0430dd477e446aa962d8ce4388ca60" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41851 } }
I20260808 02:22:48.217254 3676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:48.217288 3676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:48.217344 3676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.218108 3676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0430dd477e446aa962d8ce4388ca60" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41851 } }
I20260808 02:22:48.218197 3676 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0430dd477e446aa962d8ce4388ca60; no voters:
I20260808 02:22:48.218197 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.216301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.30.160.62:41851)
0808 02:22:48.216509 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.216516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.216540 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:22:48.216743 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.216748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.216790 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.216900 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.217718 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.217721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.217721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.217724 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.217760 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.217770 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.217826 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.217916 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.217989 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260808 02:22:48.218379 3676 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:48.218619 3676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:48.218667 3681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:48.218796 3681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0430dd477e446aa962d8ce4388ca60, State: Running, Role: LEADER
I20260808 02:22:48.218901 3681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0430dd477e446aa962d8ce4388ca60" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41851 } }
I20260808 02:22:48.219265 3682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a0430dd477e446aa962d8ce4388ca60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0430dd477e446aa962d8ce4388ca60" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41851 } } }
I20260808 02:22:48.219349 3682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:48.219501 3682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0430dd477e446aa962d8ce4388ca60. Latest consensus state: current_term: 1 leader_uuid: "2a0430dd477e446aa962d8ce4388ca60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0430dd477e446aa962d8ce4388ca60" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 41851 } } }
I20260808 02:22:48.219563 3682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:48.219647 3690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:48.220268 3690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:48.221733 3690 catalog_manager.cc:1347] Generated new cluster ID: 28cdd98b441e4efab7295b0968d17eba
I20260808 02:22:48.221792 3690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:48.234884 3690 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:48.235064 3690 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:48.247362 3690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60: Generated new TSK 0
I20260808 02:22:48.247584 3690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:48.256139 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:41851
--builtin_ntp_servers=127.30.160.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:22:48.338424 3701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:48.338616 3701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:48.338639 3701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:48.339792 3701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:48.339833 3701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:48.339859 3701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:48.341212 3701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:41851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:48.341477 3701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:48.341780 3701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:48.347389 3709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.347404 3706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.347474 3707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:48.347581 3701 server_base.cc:1034] running on GCE node
I20260808 02:22:48.347846 3701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:48.348131 3701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:48.349303 3701 hybrid_clock.cc:648] HybridClock initialized: now 1786155768349290 us; error 29 us; skew 500 ppm
I20260808 02:22:48.350592 3701 webserver.cc:460] Webserver started at http://127.30.160.1:37595/ using document root <none> and password file <none>
I20260808 02:22:48.350850 3701 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:48.350925 3701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:48.351076 3701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:48.352195 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "2ab1e90c8558142159675c195bc8d045"
server_key_iv: "b76c62dc783357483aab1c22e8d4a8cd"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.352548 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "2ab1e90c8558142159675c195bc8d045"
server_key_iv: "b76c62dc783357483aab1c22e8d4a8cd"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.353850 3701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:48.354526 3715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.354692 3701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:48.354794 3701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "2ab1e90c8558142159675c195bc8d045"
server_key_iv: "b76c62dc783357483aab1c22e8d4a8cd"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.354885 3701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:48.358198 3701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:48.358450 3701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:48.358596 3701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:48.358801 3701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:48.359112 3701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:48.359172 3701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.359210 3701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:48.359238 3701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.365013 3701 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42471
I20260808 02:22:48.365057 3832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42471 every 8 connection(s)
I20260808 02:22:48.365348 3701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:48.368405 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.365987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:37563 (local address 127.30.160.62:41851)
0808 02:22:48.366048 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.366050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.366266 (+ 216us) server_negotiation.cc:407] Connection header received
0808 02:22:48.366815 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.366817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.366825 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.366875 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.368125 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.368126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.368127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.368132 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.368148 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.368153 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.368213 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.368319 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.368337 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:22:48.368582 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.365998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:41851 (local address 127.30.160.1:37563)
0808 02:22:48.366156 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.366187 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.366701 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.366958 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.366961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.366968 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.367978 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.367979 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.368005 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.368033 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.368213 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.368218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.368221 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.368314 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.368374 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260808 02:22:48.369102 3833 heartbeater.cc:347] Connected to a master server at 127.30.160.62:41851
I20260808 02:22:48.369202 3833 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:48.369350 3833 heartbeater.cc:511] Master 127.30.160.62:41851 requested a full tablet report, sending...
I20260808 02:22:48.369817 3639 ts_manager.cc:194] Registered new tserver with Master: a2db4f2d574c455fbb21ba70c4bf6f3a (127.30.160.1:42471)
I20260808 02:22:48.371608 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3701
I20260808 02:22:48.371714 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:48.371999 31360 external_mini_cluster.cc:1428] Setting key 009bc326af723e0b734d763371e2fa6f
I20260808 02:22:48.372841 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:41851
--builtin_ntp_servers=127.30.160.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:22:48.454663 3836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:48.454870 3836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:48.454892 3836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:48.456108 3836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:48.456147 3836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:48.456173 3836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:48.457655 3836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:41851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:48.457892 3836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:48.458244 3836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:48.463145 3843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.463143 3845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.463205 3842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:48.463308 3836 server_base.cc:1034] running on GCE node
I20260808 02:22:48.463615 3836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:48.463826 3836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:48.464973 3836 hybrid_clock.cc:648] HybridClock initialized: now 1786155768464955 us; error 40 us; skew 500 ppm
I20260808 02:22:48.466248 3836 webserver.cc:460] Webserver started at http://127.30.160.2:44945/ using document root <none> and password file <none>
I20260808 02:22:48.466481 3836 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:48.466526 3836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:48.466693 3836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:48.467829 3836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "c7f4b0c153624a839a9dd402a6b67528"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "4e463adec608ffe98303927d1db1b2b1"
server_key_iv: "ef7874d2c2c5037ae7d34ae367eef5c1"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.468246 3836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "c7f4b0c153624a839a9dd402a6b67528"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "4e463adec608ffe98303927d1db1b2b1"
server_key_iv: "ef7874d2c2c5037ae7d34ae367eef5c1"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.469547 3836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:48.470238 3851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.470407 3836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:48.470513 3836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "c7f4b0c153624a839a9dd402a6b67528"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "4e463adec608ffe98303927d1db1b2b1"
server_key_iv: "ef7874d2c2c5037ae7d34ae367eef5c1"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.470588 3836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:48.473986 3836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:48.474253 3836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:48.474399 3836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:48.474622 3836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:48.474934 3836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:48.474989 3836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.475028 3836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:48.475056 3836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.480973 3836 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:33171
I20260808 02:22:48.481026 3966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:33171 every 8 connection(s)
I20260808 02:22:48.481333 3836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:48.484156 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.482058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:46825 (local address 127.30.160.62:41851)
0808 02:22:48.482123 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.482125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.482331 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:22:48.482773 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.482774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.482778 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.482846 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.483963 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.483965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.483966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.483976 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.483992 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.483996 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.484045 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.484090 (+ 45us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.484101 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:22:48.484382 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.482032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:41851 (local address 127.30.160.2:46825)
0808 02:22:48.482207 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.482247 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.482693 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.482900 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.482903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.482911 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.483774 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.483775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.483802 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.483845 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.484041 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.484043 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.484046 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.484096 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.484144 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260808 02:22:48.484926 3967 heartbeater.cc:347] Connected to a master server at 127.30.160.62:41851
I20260808 02:22:48.485013 3967 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:48.485122 3967 heartbeater.cc:511] Master 127.30.160.62:41851 requested a full tablet report, sending...
I20260808 02:22:48.485484 3639 ts_manager.cc:194] Registered new tserver with Master: c7f4b0c153624a839a9dd402a6b67528 (127.30.160.2:33171)
I20260808 02:22:48.487813 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3836
I20260808 02:22:48.487941 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:48.488125 31360 external_mini_cluster.cc:1428] Setting key 646c10f4ec22d5c3a929b857379b989b
I20260808 02:22:48.488982 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:41851
--builtin_ntp_servers=127.30.160.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:22:48.571000 3970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:48.571195 3970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:48.571216 3970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:48.572399 3970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:48.572440 3970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:48.572465 3970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:48.573897 3970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:41851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:48.574156 3970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:48.574411 3970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:48.579141 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.579155 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.579257 3976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:48.579352 3970 server_base.cc:1034] running on GCE node
I20260808 02:22:48.579593 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:48.579782 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:48.580920 3970 hybrid_clock.cc:648] HybridClock initialized: now 1786155768580920 us; error 38 us; skew 500 ppm
I20260808 02:22:48.582206 3970 webserver.cc:460] Webserver started at http://127.30.160.3:33577/ using document root <none> and password file <none>
I20260808 02:22:48.582449 3970 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:48.582494 3970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:48.582692 3970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:48.583781 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "800860289d644416ab07a87dfbc8b813"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "faec3fee54cdc4c76ebe2b9e55e8a39a"
server_key_iv: "501949ccced5bdec4096dd374a52892b"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.584200 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "800860289d644416ab07a87dfbc8b813"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "faec3fee54cdc4c76ebe2b9e55e8a39a"
server_key_iv: "501949ccced5bdec4096dd374a52892b"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.585485 3970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:48.586206 3984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.586376 3970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:48.586483 3970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "800860289d644416ab07a87dfbc8b813"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "faec3fee54cdc4c76ebe2b9e55e8a39a"
server_key_iv: "501949ccced5bdec4096dd374a52892b"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.586558 3970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:48.589888 3970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:48.590135 3970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:48.590282 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:48.590497 3970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:48.590739 3970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:48.590770 3970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.590790 3970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:48.590802 3970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.596412 3970 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:41225
I20260808 02:22:48.596490 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:41225 every 8 connection(s)
I20260808 02:22:48.596777 3970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:48.599479 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.597496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:53715 (local address 127.30.160.62:41851)
0808 02:22:48.597562 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.597564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.597762 (+ 198us) server_negotiation.cc:407] Connection header received
0808 02:22:48.598188 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.598189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.598193 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.598248 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.599286 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.599287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.599287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.599291 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.599311 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.599315 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.599361 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.599414 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.599425 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:22:48.599656 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.597470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:41851 (local address 127.30.160.3:53715)
0808 02:22:48.597643 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.597681 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.598100 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.598297 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.598300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.598308 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.599152 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.599152 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.599178 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.599206 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.599359 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.599361 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.599363 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.599413 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.599457 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260808 02:22:48.600144 4100 heartbeater.cc:347] Connected to a master server at 127.30.160.62:41851
I20260808 02:22:48.600246 4100 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:48.600399 4100 heartbeater.cc:511] Master 127.30.160.62:41851 requested a full tablet report, sending...
I20260808 02:22:48.600760 3639 ts_manager.cc:194] Registered new tserver with Master: 800860289d644416ab07a87dfbc8b813 (127.30.160.3:41225)
I20260808 02:22:48.604084 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 3970
I20260808 02:22:48.604210 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:48.604400 31360 external_mini_cluster.cc:1428] Setting key d0c615c47ee7eeed449401b47fc289b0
I20260808 02:22:48.605332 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:22:48.607613 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.606568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.30.160.62:41851)
0808 02:22:48.606639 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.606641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.606745 (+ 104us) server_negotiation.cc:407] Connection header received
0808 02:22:48.606830 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.606831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.606837 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.606894 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.607407 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.607409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.607409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.607413 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.607427 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.607430 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.607476 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.607530 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.607545 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:22:48.609127 3639 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:48.610580 3639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:48.613991 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.612685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:41225 (local address 127.0.0.1:49272)
0808 02:22:48.612799 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.612849 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.612900 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.613088 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.613095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.613102 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.613583 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.613583 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.613598 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.613620 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.613873 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.613879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.613881 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.613936 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.613946 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260808 02:22:48.614041 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.612757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.30.160.3:41225)
0808 02:22:48.612922 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.612926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.612962 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:22:48.612981 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.612982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.612988 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.613068 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.613730 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.613738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.613738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.613743 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.613777 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.613799 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.613865 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.613945 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.613961 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260808 02:22:48.615061 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.613046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33171 (local address 127.0.0.1:37958)
0808 02:22:48.613154 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.613164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.613198 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.613994 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.613995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.613998 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.614658 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.614659 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.614666 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.614678 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.614933 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.614935 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.614935 (+ 0us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.614984 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.614999 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260808 02:22:48.615087 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.613352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.30.160.2:33171)
0808 02:22:48.613547 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.613551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.613561 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:48.613590 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.613592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.613600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.613686 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.614762 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.614771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.614771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.614776 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.614809 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.614843 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.614926 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.614988 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.615004 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260808 02:22:48.614989 4033 tablet_service.cc:1467] Processing CreateTablet for tablet cf6f60a93dc3400982f9588a26f02aac (DEFAULT_TABLE table=test-table [id=cdc60f870f844f25966149aabaad42d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:48.615337 4033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6f60a93dc3400982f9588a26f02aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.615509 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.612879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42471 (local address 127.0.0.1:47246)
0808 02:22:48.613002 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.613010 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.613044 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.614380 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.614381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.614384 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.614820 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.614821 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.614828 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.614841 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.615407 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.615408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.615409 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.615439 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.615451 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260808 02:22:48.615794 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.613364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.30.160.1:42471)
0808 02:22:48.614229 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.614233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.614242 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:48.614270 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.614272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.614281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.614356 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.615245 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.615253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.615254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.615258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.615290 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.615323 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.615667 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.615691 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.615707 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":93,"threads_started":1}
I20260808 02:22:48.615723 3899 tablet_service.cc:1467] Processing CreateTablet for tablet cf6f60a93dc3400982f9588a26f02aac (DEFAULT_TABLE table=test-table [id=cdc60f870f844f25966149aabaad42d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:48.616089 3899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6f60a93dc3400982f9588a26f02aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.618121 4114 tablet_bootstrap.cc:492] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Bootstrap starting.
I20260808 02:22:48.618624 4115 tablet_bootstrap.cc:492] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Bootstrap starting.
I20260808 02:22:48.618920 4114 tablet_bootstrap.cc:654] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.618973 3765 tablet_service.cc:1467] Processing CreateTablet for tablet cf6f60a93dc3400982f9588a26f02aac (DEFAULT_TABLE table=test-table [id=cdc60f870f844f25966149aabaad42d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:48.619298 3765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6f60a93dc3400982f9588a26f02aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.619319 4114 log.cc:824] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.619349 4115 tablet_bootstrap.cc:654] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.619645 4115 log.cc:824] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.620142 4114 tablet_bootstrap.cc:492] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: No bootstrap required, opened a new log
I20260808 02:22:48.620211 4114 ts_tablet_manager.cc:1397] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:48.620471 4115 tablet_bootstrap.cc:492] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: No bootstrap required, opened a new log
I20260808 02:22:48.620558 4115 ts_tablet_manager.cc:1397] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:48.622344 4114 raft_consensus.cc:348] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.622494 4114 raft_consensus.cc:374] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.622537 4114 raft_consensus.cc:729] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800860289d644416ab07a87dfbc8b813, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.622669 4114 consensus_queue.cc:260] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.622928 4114 ts_tablet_manager.cc:1428] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:22:48.623067 4100 heartbeater.cc:503] Master 127.30.160.62:41851 was elected leader, sending a full tablet report...
I20260808 02:22:48.623219 4118 tablet_bootstrap.cc:492] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a: Bootstrap starting.
I20260808 02:22:48.623210 4115 raft_consensus.cc:348] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.623528 4115 raft_consensus.cc:374] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.623581 4115 raft_consensus.cc:729] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f4b0c153624a839a9dd402a6b67528, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.623696 4115 consensus_queue.cc:260] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.623930 4118 tablet_bootstrap.cc:654] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.623976 4115 ts_tablet_manager.cc:1428] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:22:48.624047 3967 heartbeater.cc:503] Master 127.30.160.62:41851 was elected leader, sending a full tablet report...
I20260808 02:22:48.624259 4118 log.cc:824] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.624971 4118 tablet_bootstrap.cc:492] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a: No bootstrap required, opened a new log
I20260808 02:22:48.625022 4118 ts_tablet_manager.cc:1397] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:48.626748 4118 raft_consensus.cc:348] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.626873 4118 raft_consensus.cc:374] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.626894 4118 raft_consensus.cc:729] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2db4f2d574c455fbb21ba70c4bf6f3a, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.627033 4118 consensus_queue.cc:260] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.627286 4118 ts_tablet_manager.cc:1428] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:22:48.627349 3833 heartbeater.cc:503] Master 127.30.160.62:41851 was elected leader, sending a full tablet report...
I20260808 02:22:48.716133 4119 raft_consensus.cc:482] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:48.716259 4119 raft_consensus.cc:504] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.716674 4119 leader_election.cc:288] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2db4f2d574c455fbb21ba70c4bf6f3a (127.30.160.1:42471), c7f4b0c153624a839a9dd402a6b67528 (127.30.160.2:33171)
I20260808 02:22:48.718073 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.716874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42471 (local address 127.30.160.3:33571)
0808 02:22:48.716941 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.716956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.717013 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.717179 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.717179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.717182 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.717712 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.717712 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.717719 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.717730 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.717924 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.717925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.717927 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.717979 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.717994 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:22:48.718235 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.716978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:33571 (local address 127.30.160.1:42471)
0808 02:22:48.717028 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.717031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.717039 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:48.717080 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.717081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.717087 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.717133 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.717812 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.717814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.717814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.717818 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.717842 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.717846 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.717913 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.718116 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.718132 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:22:48.718299 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.717079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:43729 (local address 127.30.160.2:33171)
0808 02:22:48.717134 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.717136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.717305 (+ 169us) server_negotiation.cc:407] Connection header received
0808 02:22:48.717345 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.717347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.717352 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.717404 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.717986 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.717988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.717988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.717993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.718027 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.718032 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.718086 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.718210 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.718227 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:48.718521 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.717006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33171 (local address 127.30.160.3:43729)
0808 02:22:48.717203 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.717212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.717269 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.717433 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.717434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.717436 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:48.717891 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:22:48.717892 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:48.717900 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:22:48.717917 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:48.718102 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:48.718103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:48.718104 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:22:48.718426 (+ 322us) client_negotiation.cc:240] Negotiation successful
0808 02:22:48.718441 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260808 02:22:48.718534 3786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6f60a93dc3400982f9588a26f02aac" candidate_uuid: "800860289d644416ab07a87dfbc8b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" is_pre_election: true
I20260808 02:22:48.718727 3786 raft_consensus.cc:2454] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800860289d644416ab07a87dfbc8b813 in term 0.
I20260808 02:22:48.718887 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6f60a93dc3400982f9588a26f02aac" candidate_uuid: "800860289d644416ab07a87dfbc8b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f4b0c153624a839a9dd402a6b67528" is_pre_election: true
I20260808 02:22:48.718966 3987 leader_election.cc:302] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800860289d644416ab07a87dfbc8b813, a2db4f2d574c455fbb21ba70c4bf6f3a; no voters:
I20260808 02:22:48.719065 3920 raft_consensus.cc:2454] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800860289d644416ab07a87dfbc8b813 in term 0.
I20260808 02:22:48.719121 4119 raft_consensus.cc:2781] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:48.719194 4119 raft_consensus.cc:482] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:48.719224 4119 raft_consensus.cc:3037] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.720083 4119 raft_consensus.cc:504] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.720220 4119 leader_election.cc:288] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [CANDIDATE]: Term 1 election: Requested vote from peers a2db4f2d574c455fbb21ba70c4bf6f3a (127.30.160.1:42471), c7f4b0c153624a839a9dd402a6b67528 (127.30.160.2:33171)
I20260808 02:22:48.720364 3786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6f60a93dc3400982f9588a26f02aac" candidate_uuid: "800860289d644416ab07a87dfbc8b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a"
I20260808 02:22:48.720464 3786 raft_consensus.cc:3037] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.720456 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6f60a93dc3400982f9588a26f02aac" candidate_uuid: "800860289d644416ab07a87dfbc8b813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f4b0c153624a839a9dd402a6b67528"
I20260808 02:22:48.720543 3920 raft_consensus.cc:3037] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.721220 3786 raft_consensus.cc:2454] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800860289d644416ab07a87dfbc8b813 in term 1.
I20260808 02:22:48.721297 3920 raft_consensus.cc:2454] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800860289d644416ab07a87dfbc8b813 in term 1.
I20260808 02:22:48.721387 3987 leader_election.cc:302] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800860289d644416ab07a87dfbc8b813, a2db4f2d574c455fbb21ba70c4bf6f3a; no voters:
I20260808 02:22:48.721467 4119 raft_consensus.cc:2781] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:48.721599 4119 raft_consensus.cc:686] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 1 LEADER]: Becoming Leader. State: Replica: 800860289d644416ab07a87dfbc8b813, State: Running, Role: LEADER
I20260808 02:22:48.721705 4119 consensus_queue.cc:237] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } }
I20260808 02:22:48.722518 3639 catalog_manager.cc:5495] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 reported cstate change: term changed from 0 to 1, leader changed from <none> to 800860289d644416ab07a87dfbc8b813 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "800860289d644416ab07a87dfbc8b813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800860289d644416ab07a87dfbc8b813" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:48.729809 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.728916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.30.160.3:41225)
0808 02:22:48.728955 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.728957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.729052 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:22:48.729120 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.729121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.729130 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.729176 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.729597 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.729598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.729598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.729601 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.729612 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.729615 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.729663 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.729700 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.729709 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:22:48.731391 3786 raft_consensus.cc:1261] T cf6f60a93dc3400982f9588a26f02aac P a2db4f2d574c455fbb21ba70c4bf6f3a [term 1 FOLLOWER]: Refusing update from remote peer 800860289d644416ab07a87dfbc8b813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:48.731392 3920 raft_consensus.cc:1261] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 1 FOLLOWER]: Refusing update from remote peer 800860289d644416ab07a87dfbc8b813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:48.731726 4124 consensus_queue.cc:1035] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7f4b0c153624a839a9dd402a6b67528" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:22:48.731849 3968 tablet.cc:2369] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:48.731848 4119 consensus_queue.cc:1035] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2db4f2d574c455fbb21ba70c4bf6f3a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:48.734642 4129 mvcc.cc:205] Tried to move back new op lower bound from 7316094028721573888 to 7316094028684824576. Current Snapshot: MvccSnapshot[applied={T|T < 7316094028721573888 or (T in {7316094028721573888})}]
I20260808 02:22:48.736272 3639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:48.736368 3639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:48.737131 3639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a0430dd477e446aa962d8ce4388ca60: Sending DeleteTablet for 3 replicas of tablet cf6f60a93dc3400982f9588a26f02aac
I20260808 02:22:48.737532 4033 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6f60a93dc3400982f9588a26f02aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:48 UTC) from {username='slave'} at 127.0.0.1:49272
I20260808 02:22:48.737577 3899 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6f60a93dc3400982f9588a26f02aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:48 UTC) from {username='slave'} at 127.0.0.1:37958
I20260808 02:22:48.737636 3765 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6f60a93dc3400982f9588a26f02aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:48 UTC) from {username='slave'} at 127.0.0.1:47246
I20260808 02:22:48.737882 4139 tablet_replica.cc:330] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: stopping tablet replica
I20260808 02:22:48.737910 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3701
I20260808 02:22:48.744608 4139 raft_consensus.cc:2229] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:48.744750 4139 raft_consensus.cc:2258] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:48.745201 4139 ts_tablet_manager.cc:1905] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:48.746743 4139 ts_tablet_manager.cc:1918] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:48.746843 4139 log.cc:1196] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/cf6f60a93dc3400982f9588a26f02aac
I20260808 02:22:48.747242 4139 ts_tablet_manager.cc:1939] T cf6f60a93dc3400982f9588a26f02aac P c7f4b0c153624a839a9dd402a6b67528: Deleting consensus metadata
I20260808 02:22:48.747663 3623 catalog_manager.cc:4841] TS c7f4b0c153624a839a9dd402a6b67528 (127.30.160.2:33171): tablet cf6f60a93dc3400982f9588a26f02aac (table test-table [id=cdc60f870f844f25966149aabaad42d6]) successfully deleted
I20260808 02:22:48.748010 4138 tablet_replica.cc:330] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: stopping tablet replica
I20260808 02:22:48.748126 4138 raft_consensus.cc:2229] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:48.748255 4138 raft_consensus.cc:2258] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:48.748657 4138 ts_tablet_manager.cc:1905] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:22:48.749898 3623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:42471 (error 108)
I20260808 02:22:48.750001 4138 ts_tablet_manager.cc:1918] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:48.750068 4138 log.cc:1196] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/cf6f60a93dc3400982f9588a26f02aac
I20260808 02:22:48.750080 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3836
I20260808 02:22:48.750330 4138 ts_tablet_manager.cc:1939] T cf6f60a93dc3400982f9588a26f02aac P 800860289d644416ab07a87dfbc8b813: Deleting consensus metadata
I20260808 02:22:48.751099 3623 catalog_manager.cc:4841] TS 800860289d644416ab07a87dfbc8b813 (127.30.160.3:41225): tablet cf6f60a93dc3400982f9588a26f02aac (table test-table [id=cdc60f870f844f25966149aabaad42d6]) successfully deleted
I20260808 02:22:48.751166 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.751048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42471 (local address 127.0.0.1:47252)
0808 02:22:48.751092 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.751136 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260808 02:22:48.751266 3623 catalog_manager.cc:4584] TS a2db4f2d574c455fbb21ba70c4bf6f3a (127.30.160.1:42471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf6f60a93dc3400982f9588a26f02aac: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42471: connect: Connection refused (error 111)
I20260808 02:22:48.757745 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3970
I20260808 02:22:48.762677 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 3606
2026-08-08T02:22:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (689 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 02:22:48.783339 31360 mini_oidc.cc:152] Starting JWKS server
I20260808 02:22:48.784598 31360 webserver.cc:460] Webserver started at https://127.0.0.1:36769/ using document root <none> and password file <none>
I20260808 02:22:48.784722 31360 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:22:48.784909 31360 webserver.cc:460] Webserver started at http://127.0.0.1:35095/ using document root <none> and password file <none>
2026-08-08T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:48Z Disabled control of system clock
I20260808 02:22:48.793486 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37317
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:37317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36769/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/testchainca.cert with env {}
W20260808 02:22:48.875530 4147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:48.875725 4147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:48.875746 4147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:48.876852 4147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:48.876888 4147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:48.876900 4147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:48.876914 4147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:48.876925 4147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:22:48.876937 4147 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:22:48.876948 4147 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36769/jwks.json
I20260808 02:22:48.878501 4147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:37317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36769/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:48.878695 4147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:48.878877 4147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:48.883558 4152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.883594 4153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:48.883564 4155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:48.883762 4147 server_base.cc:1034] running on GCE node
I20260808 02:22:48.884045 4147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:48.884315 4147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:48.885465 4147 hybrid_clock.cc:648] HybridClock initialized: now 1786155768885445 us; error 42 us; skew 500 ppm
I20260808 02:22:48.886683 4147 webserver.cc:460] Webserver started at http://127.30.160.62:36355/ using document root <none> and password file <none>
I20260808 02:22:48.886896 4147 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:48.886937 4147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:48.887024 4147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:48.888087 4147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "f09782a8197b4669903204d764391fa7"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "69fdfc612b464c1278fe1e77a9f809e8"
server_key_iv: "6db2fd9864e4c8b81efeaac96a55098e"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.888392 4147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "f09782a8197b4669903204d764391fa7"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "69fdfc612b464c1278fe1e77a9f809e8"
server_key_iv: "6db2fd9864e4c8b81efeaac96a55098e"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.889601 4147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:48.890257 4161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:48.890436 4147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:48.890528 4147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "f09782a8197b4669903204d764391fa7"
format_stamp: "Formatted at 2026-08-08 02:22:48 on dist-test-slave-zkpd"
server_key: "69fdfc612b464c1278fe1e77a9f809e8"
server_key_iv: "6db2fd9864e4c8b81efeaac96a55098e"
server_key_version: "encryptionkey@0"
I20260808 02:22:48.890611 4147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:48.908797 4147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:48.909088 4147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:48.909238 4147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:48.913242 4147 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:37317
I20260808 02:22:48.913272 4213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:37317 every 8 connection(s)
I20260808 02:22:48.913594 4147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:48.914534 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:48.917009 4214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7: Bootstrap starting.
I20260808 02:22:48.917598 4214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:48.917898 4214 log.cc:824] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:48.918000 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4147
I20260808 02:22:48.918115 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:48.918316 31360 external_mini_cluster.cc:1428] Setting key 43d7d64b016c663852d4345d83d223c2
I20260808 02:22:48.918697 4214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7: No bootstrap required, opened a new log
I20260808 02:22:48.921586 4214 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09782a8197b4669903204d764391fa7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37317 } }
I20260808 02:22:48.921746 4214 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:48.921775 4214 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09782a8197b4669903204d764391fa7, State: Initialized, Role: FOLLOWER
I20260808 02:22:48.921890 4214 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09782a8197b4669903204d764391fa7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37317 } }
I20260808 02:22:48.921979 4214 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:48.922027 4214 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:48.922070 4214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:48.922614 4214 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09782a8197b4669903204d764391fa7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37317 } }
I20260808 02:22:48.922700 4214 leader_election.cc:302] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09782a8197b4669903204d764391fa7; no voters:
I20260808 02:22:48.922885 4214 leader_election.cc:288] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:48.922912 4219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:48.923017 4219 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [term 1 LEADER]: Becoming Leader. State: Replica: f09782a8197b4669903204d764391fa7, State: Running, Role: LEADER
I20260808 02:22:48.923103 4219 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09782a8197b4669903204d764391fa7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37317 } }
I20260808 02:22:48.923166 4214 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:48.923436 4221 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09782a8197b4669903204d764391fa7. Latest consensus state: current_term: 1 leader_uuid: "f09782a8197b4669903204d764391fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09782a8197b4669903204d764391fa7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37317 } } }
I20260808 02:22:48.923671 4221 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:48.923610 4220 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09782a8197b4669903204d764391fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09782a8197b4669903204d764391fa7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37317 } } }
I20260808 02:22:48.923867 4220 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:48.924327 4226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:48.924851 4226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:48.925355 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.918801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.30.160.62:37317)
0808 02:22:48.919016 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.919026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.919060 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:22:48.920087 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.920095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.920170 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.920310 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:48.920518 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.921886 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.922800 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.923417 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.923561 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.924293 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:48.924297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:48.924298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:48.924522 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:48.924567 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:48.924583 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:48.924930 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:48.924964 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 02:22:48.925050 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260808 02:22:48.926182 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:48.928776 4226 catalog_manager.cc:1347] Generated new cluster ID: 813fdaf4344f4ed19e6bc4076e4d8afd
I20260808 02:22:48.928834 4226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:48.933925 4226 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:48.934581 4226 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:48.938634 4226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09782a8197b4669903204d764391fa7: Generated new TSK 0
I20260808 02:22:48.938792 4226 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:22:48.970553 4245 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 02:22:48.970707 4245 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260808 02:22:48.971151 4244 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:48.948255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37317 (local address 127.0.0.1:50072)
0808 02:22:48.948466 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:48.948477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:48.948548 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:48.948728 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:48.948729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:48.948731 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:22:48.948821 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.948823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.949455 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.949456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.950405 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:48.950407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.950756 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.950758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:48.950832 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.950857 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:22:48.970789 (+ 19932us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:22:48.970904 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.160.62:37317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260808 02:22:48.971500 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:48.948368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.30.160.62:37317)
0808 02:22:48.948408 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:48.948410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:48.948546 (+ 136us) server_negotiation.cc:407] Connection header received
0808 02:22:48.948638 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:48.948639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:48.948648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:48.948705 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:22:48.948887 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.949365 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.950492 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:48.950670 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:48.950719 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:48.950963 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:22:48.970622 (+ 19659us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:22:48.971412 (+ 790us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 02:22:48.971572 4218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.160.62:37317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 02:22:48.971653 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4147
2026-08-08T02:22:48Z chronyd exiting
I20260808 02:22:49.009335 31360 test_util.cc:175] -----------------------------------------------
I20260808 02:22:49.009436 31360 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155746594248-31360-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 02:22:49.010226 31360 mini_oidc.cc:152] Starting JWKS server
I20260808 02:22:49.010929 31360 webserver.cc:460] Webserver started at https://127.0.0.1:42525/ using document root <none> and password file <none>
I20260808 02:22:49.010998 31360 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:22:49.011183 31360 webserver.cc:460] Webserver started at http://127.0.0.1:42227/ using document root <none> and password file <none>
2026-08-08T02:22:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:49Z Disabled control of system clock
I20260808 02:22:49.019672 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45757
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:45757
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42525/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/testchainca.cert with env {}
W20260808 02:22:49.102993 4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:49.103186 4251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:49.103209 4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:49.104343 4251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:49.104380 4251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:49.104393 4251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:49.104405 4251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:49.104417 4251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:22:49.104429 4251 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:22:49.104440 4251 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42525/jwks.json
I20260808 02:22:49.105909 4251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:45757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42525/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:49.106105 4251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:49.106261 4251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:22:49.111131 4251 server_base.cc:1034] running on GCE node
W20260808 02:22:49.111078 4260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:49.111104 4258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:49.111091 4256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:49.111503 4251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:49.111811 4251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:49.112989 4251 hybrid_clock.cc:648] HybridClock initialized: now 1786155769112989 us; error 44 us; skew 500 ppm
I20260808 02:22:49.114320 4251 webserver.cc:460] Webserver started at http://127.30.160.62:34177/ using document root <none> and password file <none>
I20260808 02:22:49.114568 4251 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:49.114639 4251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:49.114789 4251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:49.116189 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "8aa0deb3df404e50a263ec281a408361"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "af89b1bd2f7680921bd4f9bdd738d4e2"
server_key_iv: "b09024e734592b3b210f20128af7223f"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.116680 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "8aa0deb3df404e50a263ec281a408361"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "af89b1bd2f7680921bd4f9bdd738d4e2"
server_key_iv: "b09024e734592b3b210f20128af7223f"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.118433 4251 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:49.119184 4266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:49.119369 4251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:49.119470 4251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "8aa0deb3df404e50a263ec281a408361"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "af89b1bd2f7680921bd4f9bdd738d4e2"
server_key_iv: "b09024e734592b3b210f20128af7223f"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.119558 4251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:49.145196 4251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:49.145526 4251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:49.145687 4251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:49.149626 4251 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:45757
I20260808 02:22:49.149662 4318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:45757 every 8 connection(s)
I20260808 02:22:49.150000 4251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:49.150674 4319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:49.153369 4319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361: Bootstrap starting.
I20260808 02:22:49.153968 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4251
I20260808 02:22:49.154021 4319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:49.154093 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:49.154299 31360 external_mini_cluster.cc:1428] Setting key 85a39b97055caab831fed397fd12fec8
I20260808 02:22:49.154361 4319 log.cc:824] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:49.155251 4319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361: No bootstrap required, opened a new log
I20260808 02:22:49.157859 4319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0deb3df404e50a263ec281a408361" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45757 } }
I20260808 02:22:49.158015 4319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:49.158061 4319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa0deb3df404e50a263ec281a408361, State: Initialized, Role: FOLLOWER
I20260808 02:22:49.158191 4319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0deb3df404e50a263ec281a408361" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45757 } }
I20260808 02:22:49.158288 4319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:49.158337 4319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:49.158387 4319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:49.159087 4319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0deb3df404e50a263ec281a408361" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45757 } }
I20260808 02:22:49.159198 4319 leader_election.cc:302] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aa0deb3df404e50a263ec281a408361; no voters:
I20260808 02:22:49.159376 4319 leader_election.cc:288] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:49.159475 4324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:49.159621 4319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:49.159646 4324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [term 1 LEADER]: Becoming Leader. State: Replica: 8aa0deb3df404e50a263ec281a408361, State: Running, Role: LEADER
I20260808 02:22:49.159737 4324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0deb3df404e50a263ec281a408361" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45757 } }
I20260808 02:22:49.160126 4324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aa0deb3df404e50a263ec281a408361. Latest consensus state: current_term: 1 leader_uuid: "8aa0deb3df404e50a263ec281a408361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0deb3df404e50a263ec281a408361" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45757 } } }
I20260808 02:22:49.160233 4324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:49.160578 4326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aa0deb3df404e50a263ec281a408361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0deb3df404e50a263ec281a408361" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45757 } } }
I20260808 02:22:49.160663 4326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:49.160823 4332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:49.161149 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:49.154792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.30.160.62:45757)
0808 02:22:49.154993 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:49.155003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:49.155040 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:22:49.155651 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:49.155660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:49.155734 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:49.155919 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:49.156131 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.157543 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.158585 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.159216 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.159351 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.160226 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:49.160231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:49.160231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:49.160437 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:49.160507 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:49.160523 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:49.160573 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:49.160856 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 02:22:49.160938 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260808 02:22:49.161610 4332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:49.162411 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:49.164335 4332 catalog_manager.cc:1347] Generated new cluster ID: a83ab667370044718ceb8489359a5c45
I20260808 02:22:49.164398 4332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:49.175400 4332 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:49.176136 4332 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:49.178705 4332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aa0deb3df404e50a263ec281a408361: Generated new TSK 0
I20260808 02:22:49.178870 4332 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:22:49.199878 4349 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 02:22:49.200003 4349 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:22:49.200278 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:49.185995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.30.160.62:45757)
0808 02:22:49.186065 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:49.186068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:49.186235 (+ 167us) server_negotiation.cc:407] Connection header received
0808 02:22:49.186273 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:49.186274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:49.186287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:49.186364 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:22:49.186487 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.187281 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.188099 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.188422 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.188495 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.188650 (+ 155us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:22:49.200080 (+ 11430us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:22:49.200209 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 02:22:49.200345 4323 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 02:22:49.200369 4348 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:49.185936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45757 (local address 127.0.0.1:59748)
0808 02:22:49.186122 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:49.186137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:49.186192 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:49.186369 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:49.186370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:49.186371 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:22:49.186419 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:49.186421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.187361 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.187362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:49.188011 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:49.188013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.188508 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.188509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:49.188551 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.188556 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:22:49.200190 (+ 11634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:22:49.200267 (+ 77us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.160.62:45757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260808 02:22:49.200686 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4251
2026-08-08T02:22:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 02:22:49.240437 31360 mini_oidc.cc:152] Starting JWKS server
I20260808 02:22:49.241081 31360 webserver.cc:460] Webserver started at https://127.0.0.1:35551/ using document root <none> and password file <none>
I20260808 02:22:49.241120 31360 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:22:49.241285 31360 webserver.cc:460] Webserver started at http://127.0.0.1:33483/ using document root <none> and password file <none>
2026-08-08T02:22:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:49Z Disabled control of system clock
I20260808 02:22:49.250067 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39483
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:38953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39483
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35551/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:22:49.336040 4355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:49.336226 4355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:49.336247 4355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:49.337381 4355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:49.337424 4355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:49.337437 4355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:49.337450 4355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:49.337461 4355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:22:49.337473 4355 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:22:49.337523 4355 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35551/jwks.json
I20260808 02:22:49.338905 4355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35551/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:49.339150 4355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:49.339437 4355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:49.344213 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:49.344179 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:49.344434 4355 server_base.cc:1034] running on GCE node
W20260808 02:22:49.344174 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:49.344754 4355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:49.345048 4355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:49.346235 4355 hybrid_clock.cc:648] HybridClock initialized: now 1786155769346226 us; error 35 us; skew 500 ppm
I20260808 02:22:49.347429 4355 webserver.cc:460] Webserver started at http://127.30.160.62:41481/ using document root <none> and password file <none>
I20260808 02:22:49.347672 4355 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:49.347748 4355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:49.347947 4355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:49.348968 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "042b25d7a38d45b7bd0c10b16795d360"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "4e2b6629ab428b9e0aabe61e89f049a1"
server_key_iv: "b34042d4a050c4ed6e259a6e65f36948"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.349336 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "042b25d7a38d45b7bd0c10b16795d360"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "4e2b6629ab428b9e0aabe61e89f049a1"
server_key_iv: "b34042d4a050c4ed6e259a6e65f36948"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.350597 4355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:22:49.351262 4369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:49.351464 4355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:49.351570 4355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "042b25d7a38d45b7bd0c10b16795d360"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "4e2b6629ab428b9e0aabe61e89f049a1"
server_key_iv: "b34042d4a050c4ed6e259a6e65f36948"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.351647 4355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:49.360713 4355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:49.360971 4355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:49.361115 4355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:49.365015 4355 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39483
I20260808 02:22:49.365049 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39483 every 8 connection(s)
I20260808 02:22:49.365368 4355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:49.366014 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:49.368871 4423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360: Bootstrap starting.
I20260808 02:22:49.369447 4423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:49.369729 4423 log.cc:824] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:49.370409 4423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360: No bootstrap required, opened a new log
I20260808 02:22:49.372017 4423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042b25d7a38d45b7bd0c10b16795d360" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39483 } }
I20260808 02:22:49.372146 4423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:49.372187 4423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042b25d7a38d45b7bd0c10b16795d360, State: Initialized, Role: FOLLOWER
I20260808 02:22:49.372340 4423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042b25d7a38d45b7bd0c10b16795d360" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39483 } }
I20260808 02:22:49.372404 4423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:49.372443 4423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:49.372495 4423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:49.373006 4423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042b25d7a38d45b7bd0c10b16795d360" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39483 } }
I20260808 02:22:49.373116 4423 leader_election.cc:302] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 042b25d7a38d45b7bd0c10b16795d360; no voters:
I20260808 02:22:49.373291 4423 leader_election.cc:288] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:49.373351 4426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:49.373518 4426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [term 1 LEADER]: Becoming Leader. State: Replica: 042b25d7a38d45b7bd0c10b16795d360, State: Running, Role: LEADER
I20260808 02:22:49.373548 4423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:49.373613 4426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042b25d7a38d45b7bd0c10b16795d360" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39483 } }
I20260808 02:22:49.374007 4428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042b25d7a38d45b7bd0c10b16795d360. Latest consensus state: current_term: 1 leader_uuid: "042b25d7a38d45b7bd0c10b16795d360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042b25d7a38d45b7bd0c10b16795d360" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39483 } } }
I20260808 02:22:49.374096 4428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:49.374147 4427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "042b25d7a38d45b7bd0c10b16795d360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042b25d7a38d45b7bd0c10b16795d360" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39483 } } }
I20260808 02:22:49.374215 4427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:49.374328 4434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:49.374433 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4355
I20260808 02:22:49.374559 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:49.374778 31360 external_mini_cluster.cc:1428] Setting key 64014c038168a1b42081cc34a3da638b
I20260808 02:22:49.375037 4434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:49.377599 4434 catalog_manager.cc:1347] Generated new cluster ID: 0c8284bdc08a4c08b9e9f409482a6b8b
I20260808 02:22:49.377671 4434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:49.380784 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:49.375397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.30.160.62:39483)
0808 02:22:49.375609 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:49.375618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:49.376017 (+ 399us) server_negotiation.cc:407] Connection header received
0808 02:22:49.376164 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:49.376171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:49.376258 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:49.376411 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:49.376634 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.377488 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.378623 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.379235 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.379359 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.380094 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:49.380097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:49.380098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:49.380292 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:49.380339 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:49.380352 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:49.380412 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:49.380518 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:22:49.380588 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260808 02:22:49.381583 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:49.383811 4434 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:49.384480 4434 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:49.389283 4434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 042b25d7a38d45b7bd0c10b16795d360: Generated new TSK 0
I20260808 02:22:49.389448 4434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:49.399922 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:49.394863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.30.160.62:39483)
0808 02:22:49.394922 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:49.394924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:49.396061 (+ 1137us) server_negotiation.cc:407] Connection header received
0808 02:22:49.396132 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:49.396133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:49.396143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:49.396198 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:22:49.396326 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.396793 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.398007 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.398188 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.398239 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.398430 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:22:49.399699 (+ 1269us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:22:49.399825 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 02:22:49.400048 4453 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:49.394796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39483 (local address 127.0.0.1:55464)
0808 02:22:49.395990 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:49.396002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:49.396065 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:49.396209 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:49.396210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:49.396211 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:22:49.396269 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:49.396270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.396892 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.396893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:49.397922 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:49.397925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.398264 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.398266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:49.398336 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.398342 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:22:49.399801 (+ 1459us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:22:49.399943 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.160.62:39483: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":84,"threads_started":1}
I20260808 02:22:49.400411 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4355
E20260808 02:22:49.411442 4454 webserver.cc:533] Webserver: sslize failed:
E20260808 02:22:49.411509 4454 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T02:22:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (198 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T02:22:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:49Z Disabled control of system clock
I20260808 02:22:49.447244 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:43571
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:43571
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/testchainca.cert with env {}
W20260808 02:22:49.531769 4458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:49.532047 4458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:49.532086 4458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:49.533246 4458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:49.533316 4458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:49.533340 4458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:49.533361 4458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:49.533389 4458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:22:49.533414 4458 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:22:49.533443 4458 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 02:22:49.534934 4458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:43571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:43571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:49.535207 4458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:49.535415 4458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:49.540180 4463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:49.540309 4458 server_base.cc:1034] running on GCE node
W20260808 02:22:49.540194 4464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:49.540374 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:49.540647 4458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:49.540962 4458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:49.542127 4458 hybrid_clock.cc:648] HybridClock initialized: now 1786155769542111 us; error 47 us; skew 500 ppm
I20260808 02:22:49.543351 4458 webserver.cc:460] Webserver started at http://127.30.160.62:33385/ using document root <none> and password file <none>
I20260808 02:22:49.543610 4458 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:49.543682 4458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:49.543841 4458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:49.544867 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "b5a956782bfe407b9b06ab8475c5d7cc"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "d3149130b29dcfe9d5e26ad631d0dad4"
server_key_iv: "61a686c06b7600d6724454892848c338"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.545229 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "b5a956782bfe407b9b06ab8475c5d7cc"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "d3149130b29dcfe9d5e26ad631d0dad4"
server_key_iv: "61a686c06b7600d6724454892848c338"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.546509 4458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:49.547260 4472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:49.547446 4458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:49.547554 4458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "b5a956782bfe407b9b06ab8475c5d7cc"
format_stamp: "Formatted at 2026-08-08 02:22:49 on dist-test-slave-zkpd"
server_key: "d3149130b29dcfe9d5e26ad631d0dad4"
server_key_iv: "61a686c06b7600d6724454892848c338"
server_key_version: "encryptionkey@0"
I20260808 02:22:49.547628 4458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:49.560266 4458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:49.560549 4458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:49.560698 4458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:49.564647 4458 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:43571
I20260808 02:22:49.564678 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:43571 every 8 connection(s)
I20260808 02:22:49.565002 4458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:49.565934 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:49.568186 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc: Bootstrap starting.
I20260808 02:22:49.568781 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:49.569015 4526 log.cc:824] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:49.569619 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc: No bootstrap required, opened a new log
I20260808 02:22:49.571062 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43571 } }
I20260808 02:22:49.571185 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:49.571208 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a956782bfe407b9b06ab8475c5d7cc, State: Initialized, Role: FOLLOWER
I20260808 02:22:49.571300 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43571 } }
I20260808 02:22:49.571362 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:49.571380 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:49.571405 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:49.571666 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4458
I20260808 02:22:49.571802 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:49.571995 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43571 } }
I20260808 02:22:49.572031 31360 external_mini_cluster.cc:1428] Setting key f93ebb1a98b7e5c3ffc840fc1bfaf0fe
I20260808 02:22:49.572082 4526 leader_election.cc:302] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a956782bfe407b9b06ab8475c5d7cc; no voters:
I20260808 02:22:49.572255 4526 leader_election.cc:288] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:49.572291 4529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:49.572397 4529 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [term 1 LEADER]: Becoming Leader. State: Replica: b5a956782bfe407b9b06ab8475c5d7cc, State: Running, Role: LEADER
I20260808 02:22:49.572480 4529 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43571 } }
I20260808 02:22:49.572516 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:49.572764 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43571 } } }
I20260808 02:22:49.572853 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:49.572980 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a956782bfe407b9b06ab8475c5d7cc. Latest consensus state: current_term: 1 leader_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a956782bfe407b9b06ab8475c5d7cc" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43571 } } }
I20260808 02:22:49.573053 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:49.573122 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:49.574218 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:49.576058 4535 catalog_manager.cc:1347] Generated new cluster ID: d988aa5b36e64725a898278620fc97e3
I20260808 02:22:49.576115 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:49.582729 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:49.572481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.30.160.62:43571)
0808 02:22:49.572704 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:49.572712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:49.573344 (+ 632us) server_negotiation.cc:407] Connection header received
0808 02:22:49.573516 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:49.573522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:49.573590 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:49.573752 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:49.574024 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.575301 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.580574 (+ 5273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.581222 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.581346 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.582025 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:49.582029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:49.582029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:49.582246 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:49.582296 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:49.582310 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:49.582383 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:49.582449 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 02:22:49.582524 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260808 02:22:49.583757 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:49.598766 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:49.599462 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:49.612672 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a956782bfe407b9b06ab8475c5d7cc: Generated new TSK 0
I20260808 02:22:49.612864 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:49.685117 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:49.624989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.30.160.62:43571)
0808 02:22:49.625058 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:49.625060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:49.625197 (+ 137us) server_negotiation.cc:407] Connection header received
0808 02:22:49.625259 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:49.625260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:49.625273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:49.625346 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:22:49.625476 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.626256 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.627029 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.627351 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.627423 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.627585 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:22:49.684819 (+ 57234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:22:49.685029 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46786: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 02:22:49.685204 4533 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46786: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 02:22:49.685213 4556 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:49.624927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:43571 (local address 127.0.0.1:46786)
0808 02:22:49.625120 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:49.625129 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:49.625183 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:49.625351 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:49.625351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:49.625353 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:22:49.625407 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:49.625408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.626349 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.626350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:49.626941 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:49.626943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:49.627435 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:49.627437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:49.627477 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:49.627481 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:22:49.684970 (+ 57489us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:22:49.685095 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.160.62:43571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260808 02:22:49.685412 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4458
2026-08-08T02:22:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (266 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:49 dist-test-slave-zkpd krb5kdc[4568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:49 dist-test-slave-zkpd krb5kdc[4568](info): set up 2 sockets
Aug 08 02:22:49 dist-test-slave-zkpd krb5kdc[4568](info): commencing operation
krb5kdc: starting...
W20260808 02:22:50.928512 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:22:51.050268 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:43341
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:51.134842 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:51.135076 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:51.135115 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:51.136327 4583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:51.136385 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:51.136410 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:51.136440 4583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:51.136468 4583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:51.137899 4583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:43341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:51.138177 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:51.138417 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:51.143149 4588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:51.143180 4589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.143200 4583 server_base.cc:1034] running on GCE node
W20260808 02:22:51.143146 4591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.143548 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:51.143958 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:51.145135 4583 hybrid_clock.cc:648] HybridClock initialized: now 1786155771145120 us; error 45 us; skew 500 ppm
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.150426 4583 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:22:51.150913 4583 webserver.cc:460] Webserver started at http://127.30.160.62:43099/ using document root <none> and password file <none>
I20260808 02:22:51.151139 4583 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:51.151207 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:51.151367 4583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:51.152449 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "31359e6e76e84fce9855475acc3f732d"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "81a8df58274aa70c5da82887509e4bef"
server_key_iv: "2b7a151d157aaca3904e2c1b1ad7a16e"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.152828 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "31359e6e76e84fce9855475acc3f732d"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "81a8df58274aa70c5da82887509e4bef"
server_key_iv: "2b7a151d157aaca3904e2c1b1ad7a16e"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.154148 4583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:51.154899 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.155081 4583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:51.155187 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "31359e6e76e84fce9855475acc3f732d"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "81a8df58274aa70c5da82887509e4bef"
server_key_iv: "2b7a151d157aaca3904e2c1b1ad7a16e"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.155261 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:51.170513 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:51.172412 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:51.172573 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:51.176692 4583 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:43341
I20260808 02:22:51.176724 4655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:43341 every 8 connection(s)
I20260808 02:22:51.177062 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:51.177891 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:51.180171 4656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d: Bootstrap starting.
I20260808 02:22:51.180734 4656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:51.181037 4656 log.cc:824] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:51.181702 4656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d: No bootstrap required, opened a new log
I20260808 02:22:51.183319 4656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31359e6e76e84fce9855475acc3f732d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43341 } }
I20260808 02:22:51.183454 4656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:51.183476 4656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31359e6e76e84fce9855475acc3f732d, State: Initialized, Role: FOLLOWER
I20260808 02:22:51.183669 4656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31359e6e76e84fce9855475acc3f732d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43341 } }
I20260808 02:22:51.183753 4656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:51.183796 4656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:51.183854 4656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:51.184422 4656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31359e6e76e84fce9855475acc3f732d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43341 } }
I20260808 02:22:51.184535 4656 leader_election.cc:302] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31359e6e76e84fce9855475acc3f732d; no voters:
I20260808 02:22:51.184700 4656 leader_election.cc:288] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:51.184741 4659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:51.184971 4656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:51.185081 4659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [term 1 LEADER]: Becoming Leader. State: Replica: 31359e6e76e84fce9855475acc3f732d, State: Running, Role: LEADER
I20260808 02:22:51.185173 4659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31359e6e76e84fce9855475acc3f732d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43341 } }
I20260808 02:22:51.185359 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4583
I20260808 02:22:51.185482 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:51.185606 4662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31359e6e76e84fce9855475acc3f732d. Latest consensus state: current_term: 1 leader_uuid: "31359e6e76e84fce9855475acc3f732d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31359e6e76e84fce9855475acc3f732d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43341 } } }
I20260808 02:22:51.185710 31360 external_mini_cluster.cc:1428] Setting key ab82f5720d608d26778202ad7ab461c5
I20260808 02:22:51.185626 4660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31359e6e76e84fce9855475acc3f732d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31359e6e76e84fce9855475acc3f732d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 43341 } } }
I20260808 02:22:51.185788 4660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:51.186010 4662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:51.186456 4664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:51.187034 4664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:51.188764 4664 catalog_manager.cc:1347] Generated new cluster ID: 9a16b9ce48444ec997a56425508fab54
I20260808 02:22:51.188818 4664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:51.196816 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.186409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.30.160.62:43341)
0808 02:22:51.186565 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.186575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.186610 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:22:51.186700 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.186705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.186745 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.186930 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.187740 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.189062 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.189836 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.190252 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.190410 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.192885 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.192889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.192889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.192901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.195112 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.195510 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.195513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.195513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.195603 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.195822 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.195823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.195823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.196146 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.196309 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.196436 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.196606 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260808 02:22:51.200892 4664 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:51.201370 4664 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:51.210681 4664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31359e6e76e84fce9855475acc3f732d: Generated new TSK 0
I20260808 02:22:51.210861 4664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:22:51.242929 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:43341
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:51.337679 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:51.337877 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:51.337900 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:51.339046 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:51.339087 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:51.339111 4684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:51.340591 4684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:51.340837 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:51.341071 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:51.346230 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.346400 4684 server_base.cc:1034] running on GCE node
W20260808 02:22:51.346242 4692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:51.346273 4690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.346695 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:51.346957 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:51.348143 4684 hybrid_clock.cc:648] HybridClock initialized: now 1786155771348102 us; error 45 us; skew 500 ppm
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.350654 4684 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:22:51.351084 4684 webserver.cc:460] Webserver started at http://127.30.160.1:39305/ using document root <none> and password file <none>
I20260808 02:22:51.351336 4684 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:51.351406 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:51.351559 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:51.352631 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "ec5da211f7a44280b629bebdf46afc1c"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "81d2adfc627642b62cbfb84c0a6e840e"
server_key_iv: "24b0a680fbe6ed1c3856e09502626ba4"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.352999 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "ec5da211f7a44280b629bebdf46afc1c"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "81d2adfc627642b62cbfb84c0a6e840e"
server_key_iv: "24b0a680fbe6ed1c3856e09502626ba4"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.354334 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:22:51.355051 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.355223 4684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:51.355329 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "ec5da211f7a44280b629bebdf46afc1c"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "81d2adfc627642b62cbfb84c0a6e840e"
server_key_iv: "24b0a680fbe6ed1c3856e09502626ba4"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.355404 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:51.369697 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:51.371398 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:51.371555 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:51.371769 4684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:51.372161 4684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:51.372212 4684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.372252 4684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:51.372283 4684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.378060 4684 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:44993
I20260808 02:22:51.378159 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:44993 every 8 connection(s)
I20260808 02:22:51.378460 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:51.378724 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4684
I20260808 02:22:51.378834 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:51.379040 31360 external_mini_cluster.cc:1428] Setting key abf887d6485c689c069592662044ae24
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
I20260808 02:22:51.388814 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.379205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:54413 (local address 127.30.160.62:43341)
0808 02:22:51.379255 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.379257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.379521 (+ 264us) server_negotiation.cc:407] Connection header received
0808 02:22:51.380661 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.380662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.380679 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.380745 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.381341 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.381954 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.383234 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.383444 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.383516 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.385759 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.385761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.385762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.385774 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.387395 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.387832 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.387833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.387833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.387916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.388157 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.388158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.388159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.388280 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.388357 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.388628 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.388713 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:22:51.388998 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.379202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:43341 (local address 127.30.160.1:54413)
0808 02:22:51.379394 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:51.379440 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:51.380553 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:51.380819 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:51.380822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:51.381072 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:51.381252 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:51.381256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.382047 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.382048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:51.383131 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:51.383134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.383539 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.383540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:51.383672 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.384228 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:51.384236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:51.385641 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:51.387497 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:51.387505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:51.387524 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:51.387750 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:51.387968 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:51.387969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:51.387970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:51.388086 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:51.388350 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:51.388358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:51.388497 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 02:22:51.388606 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:22:51.388762 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260808 02:22:51.389470 4817 heartbeater.cc:347] Connected to a master server at 127.30.160.62:43341
I20260808 02:22:51.389588 4817 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:51.389771 4817 heartbeater.cc:511] Master 127.30.160.62:43341 requested a full tablet report, sending...
I20260808 02:22:51.390353 4616 ts_manager.cc:194] Registered new tserver with Master: ec5da211f7a44280b629bebdf46afc1c (127.30.160.1:44993)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
I20260808 02:22:51.390980 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:54413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:22:51.404263 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:43341
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:51.490734 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:51.490938 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:51.490962 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:51.492197 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:51.492239 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:51.492265 4822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:51.493762 4822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:51.494004 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:51.494227 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:51.499128 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:51.499133 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.499184 4822 server_base.cc:1034] running on GCE node
W20260808 02:22:51.499133 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.499526 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:51.499786 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:51.500972 4822 hybrid_clock.cc:648] HybridClock initialized: now 1786155771500959 us; error 25 us; skew 500 ppm
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.503328 4822 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:22:51.503795 4822 webserver.cc:460] Webserver started at http://127.30.160.2:42015/ using document root <none> and password file <none>
I20260808 02:22:51.504046 4822 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:51.504117 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:51.504273 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:51.505275 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "d38c428e22b747ad846c5ec5720e9346"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "84b7d7afc24573fec5965d0965bb6633"
server_key_iv: "05bc34e4b34283bf1b5091096e70a07d"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.505640 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "d38c428e22b747ad846c5ec5720e9346"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "84b7d7afc24573fec5965d0965bb6633"
server_key_iv: "05bc34e4b34283bf1b5091096e70a07d"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.506937 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:51.507637 4837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.507810 4822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:51.507946 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "d38c428e22b747ad846c5ec5720e9346"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "84b7d7afc24573fec5965d0965bb6633"
server_key_iv: "05bc34e4b34283bf1b5091096e70a07d"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.508034 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:51.518958 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:51.520800 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:51.520974 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:51.521194 4822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:51.521495 4822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:51.521550 4822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.521586 4822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:51.521612 4822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.527590 4822 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:33459
I20260808 02:22:51.527642 4955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:33459 every 8 connection(s)
I20260808 02:22:51.528016 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:51.529055 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4822
I20260808 02:22:51.529156 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:51.529354 31360 external_mini_cluster.cc:1428] Setting key ae9dfd85e86f59d4efbc77234f914c19
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:51.538560 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.528713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:54379 (local address 127.30.160.62:43341)
0808 02:22:51.528761 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.528764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.528971 (+ 207us) server_negotiation.cc:407] Connection header received
0808 02:22:51.530127 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.530128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.530145 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.530191 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.531011 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.531501 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.532680 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.533008 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.533085 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.535672 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.535674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.535674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.535681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.537077 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.537597 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.537598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.537598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.537648 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.537933 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.537934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.537934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.538032 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.538081 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.538430 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.538496 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:51.538884 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.528661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:43341 (local address 127.30.160.2:54379)
0808 02:22:51.528836 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:51.528877 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:51.530007 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:51.530267 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:51.530271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:51.530636 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:51.530922 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:51.530929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.531597 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.531598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:51.532587 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:51.532590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.533097 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.533098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:51.533202 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.533728 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:51.533736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:51.535553 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:51.537165 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:51.537173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:51.537175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:51.537511 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:51.537715 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:51.537716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:51.537716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:51.537862 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:51.538100 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:51.538107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:51.538280 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 02:22:51.538415 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 02:22:51.538641 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
I20260808 02:22:51.539412 4956 heartbeater.cc:347] Connected to a master server at 127.30.160.62:43341
I20260808 02:22:51.539526 4956 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:51.539700 4956 heartbeater.cc:511] Master 127.30.160.62:43341 requested a full tablet report, sending...
I20260808 02:22:51.540160 4616 ts_manager.cc:194] Registered new tserver with Master: d38c428e22b747ad846c5ec5720e9346 (127.30.160.2:33459)
I20260808 02:22:51.540545 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:54379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:22:51.556473 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:43341
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:51.638522 4961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:51.638715 4961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:51.638737 4961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:51.639878 4961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:51.639978 4961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:51.640007 4961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:51.641444 4961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:51.641673 4961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:51.641904 4961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:51.646847 4969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:51.646879 4967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:51.646876 4966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:51.647023 4961 server_base.cc:1034] running on GCE node
I20260808 02:22:51.647249 4961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:51.647480 4961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:51.648648 4961 hybrid_clock.cc:648] HybridClock initialized: now 1786155771648630 us; error 38 us; skew 500 ppm
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.651155 4961 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:22:51.651592 4961 webserver.cc:460] Webserver started at http://127.30.160.3:45293/ using document root <none> and password file <none>
I20260808 02:22:51.651764 4961 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:51.651808 4961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:51.651952 4961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:51.652897 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "d6686b1b465a4d0f8caae113a2d99ee6"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "b856af12098abfedc63b6d296aa5c660"
server_key_iv: "68c9195be3ba59d5457e450d4aaeff38"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.653191 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "d6686b1b465a4d0f8caae113a2d99ee6"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "b856af12098abfedc63b6d296aa5c660"
server_key_iv: "68c9195be3ba59d5457e450d4aaeff38"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.654471 4961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:22:51.655169 4976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.655339 4961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:51.655445 4961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "d6686b1b465a4d0f8caae113a2d99ee6"
format_stamp: "Formatted at 2026-08-08 02:22:51 on dist-test-slave-zkpd"
server_key: "b856af12098abfedc63b6d296aa5c660"
server_key_iv: "68c9195be3ba59d5457e450d4aaeff38"
server_key_version: "encryptionkey@0"
I20260808 02:22:51.655519 4961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:51.672077 4961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:51.674391 4961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:51.674552 4961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:51.674827 4961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:51.675123 4961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:51.675176 4961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.675216 4961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:51.675248 4961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:51.680984 4961 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:44245
I20260808 02:22:51.681036 5097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:44245 every 8 connection(s)
I20260808 02:22:51.681326 4961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:51.681670 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 4961
I20260808 02:22:51.681761 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:51.681953 31360 external_mini_cluster.cc:1428] Setting key 927c853823a095c7ec114703408fec4a
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:22:51.690734 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.682077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:53705 (local address 127.30.160.62:43341)
0808 02:22:51.682152 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.682154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.682365 (+ 211us) server_negotiation.cc:407] Connection header received
0808 02:22:51.683486 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.683487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.683501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.683567 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.684268 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.684823 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.685883 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.686179 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.686249 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.688475 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.688477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.688477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.688484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.689664 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.690017 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.690018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.690018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.690069 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.690239 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.690240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.690240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.690337 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.690387 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.690610 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.690670 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:22:51.690937 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.682043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:43341 (local address 127.30.160.3:53705)
0808 02:22:51.682228 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:51.682274 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:51.683369 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:51.683644 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:51.683649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:51.683968 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:51.684160 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:51.684164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.684924 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.684927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:51.685790 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:51.685793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.686257 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.686259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:51.686391 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.686929 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:51.686937 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:51.688392 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:51.689735 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:51.689741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:51.689743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:51.689951 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:51.690110 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:51.690111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:51.690111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:51.690191 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:51.690394 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:51.690399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:51.690503 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 02:22:51.690583 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:22:51.690695 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260808 02:22:51.691476 5098 heartbeater.cc:347] Connected to a master server at 127.30.160.62:43341
I20260808 02:22:51.691591 5098 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:51.691769 5098 heartbeater.cc:511] Master 127.30.160.62:43341 requested a full tablet report, sending...
I20260808 02:22:51.692150 4616 ts_manager.cc:194] Registered new tserver with Master: d6686b1b465a4d0f8caae113a2d99ee6 (127.30.160.3:44245)
I20260808 02:22:51.692523 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:53705
I20260808 02:22:51.693645 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/test-admin.keytab.
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.708290 31360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 02:22:51.710117 5105 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:51.709256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.30.160.1:44993)
0808 02:22:51.709517 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.709521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.709532 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:51.709557 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.709559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.709574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.709653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.710027 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45150: BlockingRecv error: recv got EOF from 127.0.0.1:45150 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
W20260808 02:22:51.710274 31360 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.711344 31360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.712386 31360 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:51.720162 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.712875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.30.160.1:44993)
0808 02:22:51.712930 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.712932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.713017 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:22:51.713070 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.713071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.713082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.713121 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.713487 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.714242 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.714913 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.715217 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.715275 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.717314 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.717329 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.717329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.717335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.718933 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.719318 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.719321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.719321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.719381 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.719590 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.719591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.719591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.719720 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.719926 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.719972 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.720046 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 02:22:51.721832 5105 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:51.721320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.30.160.1:44993)
0808 02:22:51.721357 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.721359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.721405 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:22:51.721462 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.721463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.721471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.721513 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.721782 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45162: BlockingRecv error: recv got EOF from 127.0.0.1:45162 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:22:51.721983 31360 init.cc:244] Error while opening credential cache '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc)
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.722910 31360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.723799 31360 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:51.730116 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.724305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.30.160.1:44993)
0808 02:22:51.724352 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.724353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.724377 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:22:51.724417 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.724418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.724426 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.724474 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.724796 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.725262 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.725916 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.726098 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.726147 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.728160 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.728161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.728161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.728166 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.729297 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.729593 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.729594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.729594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.729643 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.729785 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.729785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.729785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.729861 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.729923 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.730007 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.730062 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260808 02:22:51.731648 5105 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:51.731073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.30.160.1:44993)
0808 02:22:51.731106 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.731107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.731205 (+ 98us) server_negotiation.cc:407] Connection header received
0808 02:22:51.731277 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.731277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.731287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.731348 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.731602 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45172: BlockingRecv error: recv got EOF from 127.0.0.1:45172 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260808 02:22:51.731737 31360 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155746594248-31360-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.732638 31360 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:51.733472 31360 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:22:51.740608 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:51.733971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.30.160.1:44993)
0808 02:22:51.734021 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:51.734023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:51.734063 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:22:51.734134 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:51.734135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:51.734144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:51.734196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:51.734554 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.735000 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.736081 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:51.736373 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:51.736430 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:51.738237 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:51.738238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:51.738238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:51.738244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:51.739557 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.739846 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.739847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.739847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.739961 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:51.740165 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:51.740166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:51.740166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:51.740243 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:51.740317 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:51.740411 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:22:51.740463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:22:51.741161 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4684
I20260808 02:22:51.747538 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4822
I20260808 02:22:51.753931 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4961
I20260808 02:22:51.759436 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 4583
2026-08-08T02:22:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1894 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[5118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[5118](info): set up 2 sockets
Aug 08 02:22:51 dist-test-slave-zkpd krb5kdc[5118](info): commencing operation
krb5kdc: starting...
W20260808 02:22:52.819770 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:52 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab.
Entry for principal oryx/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab.
I20260808 02:22:52.936182 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:36979
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab
--principal=oryx/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:36979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:53.017828 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:53.018029 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:53.018052 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:53.019217 5131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:53.019261 5131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:53.019276 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:53.019327 5131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:53.019354 5131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:53.020960 5131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:36979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab
--principal=oryx/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:36979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:53.021214 5131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:53.021459 5131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:53.026371 5140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.026400 5131 server_base.cc:1034] running on GCE node
W20260808 02:22:53.026409 5137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.026371 5138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.026875 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:53.027158 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:53.028316 5131 hybrid_clock.cc:648] HybridClock initialized: now 1786155773028305 us; error 34 us; skew 500 ppm
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:53.030912 5131 init.cc:377] Logged in from keytab as oryx/127.30.160.62@KRBTEST.COM (short username oryx)
I20260808 02:22:53.031299 5131 webserver.cc:460] Webserver started at http://127.30.160.62:36021/ using document root <none> and password file <none>
I20260808 02:22:53.031520 5131 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.031564 5131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.031728 5131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:53.032832 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "bbceee63e42341518b449bd002318f22"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "4f9092ffe3bd736862c713a840082682"
server_key_iv: "744a788c9f653eed758327eaf895d2cd"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.033200 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "bbceee63e42341518b449bd002318f22"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "4f9092ffe3bd736862c713a840082682"
server_key_iv: "744a788c9f653eed758327eaf895d2cd"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.034541 5131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:53.035307 5147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.035497 5131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:53.035596 5131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "bbceee63e42341518b449bd002318f22"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "4f9092ffe3bd736862c713a840082682"
server_key_iv: "744a788c9f653eed758327eaf895d2cd"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.035678 5131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.054320 5131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:53.056741 5131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:53.056905 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:53.060708 5131 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:36979
I20260808 02:22:53.060783 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:36979 every 8 connection(s)
I20260808 02:22:53.061127 5131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:53.062137 5204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:53.064631 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Bootstrap starting.
I20260808 02:22:53.065265 5204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:53.065580 5204 log.cc:824] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:53.066326 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: No bootstrap required, opened a new log
I20260808 02:22:53.067925 5204 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } }
I20260808 02:22:53.068061 5204 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:53.068117 5204 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbceee63e42341518b449bd002318f22, State: Initialized, Role: FOLLOWER
I20260808 02:22:53.068264 5204 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } }
I20260808 02:22:53.068331 5204 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:53.068372 5204 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:53.068421 5204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:53.069226 5204 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } }
I20260808 02:22:53.069348 5204 leader_election.cc:302] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbceee63e42341518b449bd002318f22; no voters:
I20260808 02:22:53.069528 5204 leader_election.cc:288] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:53.069554 5207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:53.069691 5207 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 LEADER]: Becoming Leader. State: Replica: bbceee63e42341518b449bd002318f22, State: Running, Role: LEADER
I20260808 02:22:53.069782 5207 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } }
I20260808 02:22:53.069809 5204 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:53.070144 5209 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbceee63e42341518b449bd002318f22. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } }
I20260808 02:22:53.070163 5208 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } }
I20260808 02:22:53.070235 5209 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:53.070240 5208 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:53.070449 5213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:53.070825 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5131
I20260808 02:22:53.070938 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:53.071143 31360 external_mini_cluster.cc:1428] Setting key 65bab8d5c997594248ed39826a220ca8
I20260808 02:22:53.071516 5213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:53.073849 5213 catalog_manager.cc:1347] Generated new cluster ID: 15468bf169d34060b08eb4005209c309
I20260808 02:22:53.073904 5213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:53.082109 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.071672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.30.160.62:36979)
0808 02:22:53.071839 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.071850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.072206 (+ 356us) server_negotiation.cc:407] Connection header received
0808 02:22:53.072429 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.072436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.072490 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.072636 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.073523 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.074551 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.075295 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.075949 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.076091 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.078397 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.078402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.078402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.078414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.080524 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.080918 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.080920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.080921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.081011 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.081229 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.081230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.081230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.081460 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.081628 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.081704 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.081949 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260808 02:22:53.087175 5213 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:53.087666 5213 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:53.089710 5213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Generated new TSK 0
I20260808 02:22:53.089864 5213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab.
Entry for principal oryx/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab.
I20260808 02:22:53.118532 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:36979
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:53.202068 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:53.202260 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:53.202284 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:53.203409 5231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:53.203450 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:53.203476 5231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:53.204982 5231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:53.205216 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:53.205472 5231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:53.210403 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.210377 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.210388 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.210711 5231 server_base.cc:1034] running on GCE node
I20260808 02:22:53.210933 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:53.211174 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:53.212333 5231 hybrid_clock.cc:648] HybridClock initialized: now 1786155773212327 us; error 42 us; skew 500 ppm
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:53.214916 5231 init.cc:377] Logged in from keytab as oryx/127.30.160.1@KRBTEST.COM (short username oryx)
I20260808 02:22:53.215358 5231 webserver.cc:460] Webserver started at http://127.30.160.1:35417/ using document root <none> and password file <none>
I20260808 02:22:53.215552 5231 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.215592 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.215682 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:53.216735 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "3177ad2f6fcb4ef0962396eb5ad844d3"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "69adf7c25f8bcaa2110c561ef5624032"
server_key_iv: "b9a554d6f855868a5d7c7795d5262b7e"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.217046 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "3177ad2f6fcb4ef0962396eb5ad844d3"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "69adf7c25f8bcaa2110c561ef5624032"
server_key_iv: "b9a554d6f855868a5d7c7795d5262b7e"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.218267 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:22:53.218950 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.219105 5231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:53.219192 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "3177ad2f6fcb4ef0962396eb5ad844d3"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "69adf7c25f8bcaa2110c561ef5624032"
server_key_iv: "b9a554d6f855868a5d7c7795d5262b7e"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.219246 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.236941 5231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:53.238854 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:53.239034 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:53.239272 5231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:53.239578 5231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:53.239611 5231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.239665 5231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:53.239691 5231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.245508 5231 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:39219
I20260808 02:22:53.245566 5368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:39219 every 8 connection(s)
I20260808 02:22:53.245895 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:53.253664 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5231
I20260808 02:22:53.253773 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:53.254001 31360 external_mini_cluster.cc:1428] Setting key 4387dde875a1e0883b267c34df486a18
I20260808 02:22:53.255395 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.246649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:56633 (local address 127.30.160.62:36979)
0808 02:22:53.246693 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.246695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.246946 (+ 251us) server_negotiation.cc:407] Connection header received
0808 02:22:53.247476 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.247477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.247491 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.247545 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.248543 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.249047 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.250028 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.250238 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.250318 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.252387 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.252388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.252389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.252397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.253916 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.254376 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.254377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.254378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.254447 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.254741 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.254742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.254743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.254863 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.254937 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.255210 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.255296 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:22:53.255560 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.246637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:56633)
0808 02:22:53.246831 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.246866 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.247364 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.247619 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.247623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.248253 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.248435 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.248439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.249147 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.249148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.249931 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.249934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.250336 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.250337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.250454 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.250944 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.250952 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.252290 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.254007 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.254015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.254017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.254287 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.254518 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.254519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.254519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.254651 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.254940 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.254947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.255079 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.255185 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.255334 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260808 02:22:53.256022 5369 heartbeater.cc:347] Connected to a master server at 127.30.160.62:36979
I20260808 02:22:53.256125 5369 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:53.256292 5369 heartbeater.cc:511] Master 127.30.160.62:36979 requested a full tablet report, sending...
I20260808 02:22:53.256903 5165 ts_manager.cc:194] Registered new tserver with Master: 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219)
I20260808 02:22:53.257477 5165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:56633
WARNING: no policy specified for oryx/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab.
Entry for principal oryx/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab.
I20260808 02:22:53.278728 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:36979
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:53.359963 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:53.360172 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:53.360211 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:53.361336 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:53.361399 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:53.361446 5374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:53.362814 5374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:53.363061 5374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:53.363293 5374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:53.368129 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.368142 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.368328 5374 server_base.cc:1034] running on GCE node
W20260808 02:22:53.368356 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.368566 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:53.368809 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:53.369980 5374 hybrid_clock.cc:648] HybridClock initialized: now 1786155773369967 us; error 29 us; skew 500 ppm
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:53.372599 5374 init.cc:377] Logged in from keytab as oryx/127.30.160.2@KRBTEST.COM (short username oryx)
I20260808 02:22:53.373059 5374 webserver.cc:460] Webserver started at http://127.30.160.2:37361/ using document root <none> and password file <none>
I20260808 02:22:53.373291 5374 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.373364 5374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.373520 5374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:53.374544 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "1cf3c56e80e94739a449813296fb5f2e"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "7e329705cb209cf12e3e802aeef3cee4"
server_key_iv: "09a5ef7b20ab903a3f78d5fea8b7bb8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.374905 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "1cf3c56e80e94739a449813296fb5f2e"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "7e329705cb209cf12e3e802aeef3cee4"
server_key_iv: "09a5ef7b20ab903a3f78d5fea8b7bb8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.376369 5374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:53.377159 5390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.377352 5374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.377452 5374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "1cf3c56e80e94739a449813296fb5f2e"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "7e329705cb209cf12e3e802aeef3cee4"
server_key_iv: "09a5ef7b20ab903a3f78d5fea8b7bb8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.377538 5374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.389523 5374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:53.391806 5374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:53.391983 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:53.392210 5374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:53.392513 5374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:53.392565 5374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.392604 5374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:53.392633 5374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.398625 5374 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:34901
I20260808 02:22:53.398682 5513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:34901 every 8 connection(s)
I20260808 02:22:53.398972 5374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:53.403268 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5374
I20260808 02:22:53.403391 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:53.403597 31360 external_mini_cluster.cc:1428] Setting key 5418bd2fe10ab6db0414aa00c4d9e4ce
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:53.408306 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.399681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:52357 (local address 127.30.160.62:36979)
0808 02:22:53.399748 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.399750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.399940 (+ 190us) server_negotiation.cc:407] Connection header received
0808 02:22:53.400423 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.400424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.400434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.400495 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.401379 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.401843 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.402772 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.402955 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.403005 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.405494 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.405496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.405496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.405505 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.406876 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.407282 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.407283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.407283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.407353 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.407632 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.407633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.407633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.407744 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.407822 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.408134 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.408216 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:22:53.408486 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.399633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:52357)
0808 02:22:53.399799 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.399845 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.400336 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.400567 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.400570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.401123 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.401299 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.401303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.401919 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.401920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.402690 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.402692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.403027 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.403028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.403124 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.403657 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.403669 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.405391 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.406977 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.406984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.406986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.407202 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.407423 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.407424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.407424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.407538 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.407819 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.407828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.407991 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.408109 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.408263 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260808 02:22:53.408972 5514 heartbeater.cc:347] Connected to a master server at 127.30.160.62:36979
I20260808 02:22:53.409049 5514 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:53.409240 5514 heartbeater.cc:511] Master 127.30.160.62:36979 requested a full tablet report, sending...
I20260808 02:22:53.409612 5165 ts_manager.cc:194] Registered new tserver with Master: 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2:34901)
I20260808 02:22:53.410169 5165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:52357
WARNING: no policy specified for oryx/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab.
Entry for principal oryx/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab.
I20260808 02:22:53.429893 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:36979
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:53.513509 5521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:53.513711 5521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:53.513762 5521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:53.514897 5521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:53.514940 5521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:53.514999 5521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:53.516539 5521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:53.516768 5521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:53.516997 5521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:53.521754 5527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.521740 5526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.521818 5529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.521961 5521 server_base.cc:1034] running on GCE node
I20260808 02:22:53.522217 5521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:53.522480 5521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:53.523626 5521 hybrid_clock.cc:648] HybridClock initialized: now 1786155773523616 us; error 30 us; skew 500 ppm
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:53.526188 5521 init.cc:377] Logged in from keytab as oryx/127.30.160.3@KRBTEST.COM (short username oryx)
I20260808 02:22:53.526655 5521 webserver.cc:460] Webserver started at http://127.30.160.3:43081/ using document root <none> and password file <none>
I20260808 02:22:53.526873 5521 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.526916 5521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.527060 5521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:53.528119 5521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "b0c55e028888437e9969d643eff2f5de"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "b2588fd30b61899b033bbfb6562068e9"
server_key_iv: "d404ef14b9c71a5757662ff233e9e979"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.528471 5521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "b0c55e028888437e9969d643eff2f5de"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "b2588fd30b61899b033bbfb6562068e9"
server_key_iv: "d404ef14b9c71a5757662ff233e9e979"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.529743 5521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:53.530412 5536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.530578 5521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:53.530663 5521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "b0c55e028888437e9969d643eff2f5de"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "b2588fd30b61899b033bbfb6562068e9"
server_key_iv: "d404ef14b9c71a5757662ff233e9e979"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.530732 5521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.539649 5521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:53.541491 5521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:53.541644 5521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:53.541872 5521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:53.542169 5521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:53.542202 5521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.542245 5521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:53.542272 5521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.547778 5521 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:34837
I20260808 02:22:53.547874 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:34837 every 8 connection(s)
I20260808 02:22:53.548198 5521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:53.554230 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5521
I20260808 02:22:53.554354 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:22:53.554567 31360 external_mini_cluster.cc:1428] Setting key 9872a5f9214ba3b12911959c7c0a42c3
I20260808 02:22:53.557199 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.548924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:37925 (local address 127.30.160.62:36979)
0808 02:22:53.548978 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.548981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.549185 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:22:53.549642 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.549643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.549652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.549704 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.550571 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.551065 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.552068 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.552248 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.552302 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.554503 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.554505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.554505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.554512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.556092 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.556452 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.556453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.556453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.556504 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.556711 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.556712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.556712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.556809 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.556871 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.557080 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.557136 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:22:53.557370 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.548911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:37925)
0808 02:22:53.549069 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.549103 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.549560 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.549751 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.549754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.550317 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.550497 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.550501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.551143 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.551144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.551997 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.552000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.552327 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.552328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.552423 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.552936 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.552944 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.554399 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.556163 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.556169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.556171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.556387 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.556561 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.556562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.556562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.556658 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.556875 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.556879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.556988 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.557067 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.557179 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260808 02:22:53.557838 5660 heartbeater.cc:347] Connected to a master server at 127.30.160.62:36979
I20260808 02:22:53.557941 5660 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:53.558109 5660 heartbeater.cc:511] Master 127.30.160.62:36979 requested a full tablet report, sending...
I20260808 02:22:53.558522 5165 ts_manager.cc:194] Registered new tserver with Master: b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837)
I20260808 02:22:53.558971 5165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:37925
I20260808 02:22:53.566511 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.61.keytab.
Entry for principal oryx/127.30.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.61.keytab.
I20260808 02:22:53.592491 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:38335
--webserver_interface=127.30.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.61.keytab
--principal=oryx/127.30.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:36979,127.30.160.61:38335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:53.676061 5666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:53.676285 5666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:53.676324 5666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:53.677600 5666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:53.677665 5666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:53.677690 5666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:53.677717 5666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:53.677749 5666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:53.679208 5666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:36979,127.30.160.61:38335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.61.keytab
--principal=oryx/127.30.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
W20260808 02:22:53.679342 5666 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.160.62:36979,127.30.160.61:38335'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 02:22:53.679497 5666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:53.679715 5666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:53.684558 5675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.684553 5672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.684540 5673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.684808 5666 server_base.cc:1034] running on GCE node
I20260808 02:22:53.685042 5666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:53.685312 5666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:53.686480 5666 hybrid_clock.cc:648] HybridClock initialized: now 1786155773686468 us; error 28 us; skew 500 ppm
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:53.688959 5666 init.cc:377] Logged in from keytab as oryx/127.30.160.61@KRBTEST.COM (short username oryx)
I20260808 02:22:53.689422 5666 webserver.cc:460] Webserver started at http://127.30.160.61:40935/ using document root <none> and password file <none>
I20260808 02:22:53.689632 5666 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.689716 5666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.689872 5666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:53.690879 5666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/instance:
uuid: "f0f0d4a6c0144573955a055e9eb58caa"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "a0892b3d6f5d4ba1055263d5101d3f47"
server_key_iv: "66486e5b460d5c28e21ddd84441e682f"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.691229 5666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal/instance:
uuid: "f0f0d4a6c0144573955a055e9eb58caa"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "a0892b3d6f5d4ba1055263d5101d3f47"
server_key_iv: "66486e5b460d5c28e21ddd84441e682f"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.692663 5666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:53.693389 5682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.693557 5666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:53.693660 5666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
uuid: "f0f0d4a6c0144573955a055e9eb58caa"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "a0892b3d6f5d4ba1055263d5101d3f47"
server_key_iv: "66486e5b460d5c28e21ddd84441e682f"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.693743 5666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.724336 5666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:53.726715 5666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:53.726886 5666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:53.731005 5666 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.61:38335
I20260808 02:22:53.731038 5738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.61:38335 every 8 connection(s)
I20260808 02:22:53.731381 5666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
I20260808 02:22:53.732404 5739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:53.734054 5739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:53.737435 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5666
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.61@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
I20260808 02:22:53.743412 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.734754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.30.160.62:36979)
0808 02:22:53.734824 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.734826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.735056 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:22:53.735503 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.735504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.735514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.735567 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.736196 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.736718 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.737716 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.737979 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.738082 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.740542 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.740543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.740544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.740550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.741993 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.742403 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.742404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.742405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.742465 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.742725 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.742726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.742726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.742879 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.742960 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.743230 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.743321 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:22:53.743558 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.734737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33048)
0808 02:22:53.734942 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.734977 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.735422 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.735645 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.735648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.735879 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.736124 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.736129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.736796 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.736797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.737632 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.737635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.738151 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.738152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.738330 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.738851 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.738858 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.740457 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.742091 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.742099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.742101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.742325 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.742540 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.742541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.742541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.742651 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.742952 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.742958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.743093 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.743196 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.743341 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260808 02:22:53.744211 5739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:53.747174 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.737982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.30.160.61:38335)
0808 02:22:53.738307 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.738311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.738329 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:22:53.738349 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.738351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.738395 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.738466 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.739263 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.740119 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.741113 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.741604 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.741685 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.743974 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.744002 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.744002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.744009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.746005 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.746366 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.746368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.746368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.746419 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.746615 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.746616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.746616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.746782 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.746946 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.746985 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.747058 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":91,"threads_started":1}
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.61@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
I20260808 02:22:53.751705 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.744477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33590)
0808 02:22:53.744526 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.744533 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.744561 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.744797 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.744797 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.744952 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.744997 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.744998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.745969 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.745970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.746680 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.746682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.747169 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.747170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.747233 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.747644 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.747649 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.748894 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.750719 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.750720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.750720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.750883 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.751084 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.751084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.751085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.751137 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.751381 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.751382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.751418 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.751487 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.751577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 02:22:53.751796 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.744524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.30.160.61:38335)
0808 02:22:53.744656 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.744658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.744666 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:53.744685 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.744686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.744697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.744794 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.745098 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.745889 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.746746 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.747070 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.747131 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.748988 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.748989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.748990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.748996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.750621 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.750955 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.750956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.750956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.751009 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.751208 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.751210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.751210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.751314 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.751388 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.751512 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.751627 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260808 02:22:53.752991 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Bootstrap starting.
I20260808 02:22:53.753551 5739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:53.753778 5739 log.cc:824] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:53.754415 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: No bootstrap required, opened a new log
I20260808 02:22:53.755700 5739 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } }
I20260808 02:22:53.755811 5739 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:53.755832 5739 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f0d4a6c0144573955a055e9eb58caa, State: Initialized, Role: FOLLOWER
I20260808 02:22:53.756031 5739 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } }
I20260808 02:22:53.756285 5747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } } }
I20260808 02:22:53.756356 5739 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:53.756372 5747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:22:53.757916 5758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:53.757994 5758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:53.759509 5666 master_runner.cc:428] Detected that this master f0f0d4a6c0144573955a055e9eb58caa is joining an existing cluster
I20260808 02:22:53.759557 5666 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.160.61:38335
I20260808 02:22:53.767886 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.760684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.30.160.62:36979)
0808 02:22:53.760750 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.760752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.760870 (+ 118us) server_negotiation.cc:407] Connection header received
0808 02:22:53.761020 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.761022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.761031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.761097 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.761597 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.762377 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.763117 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.763366 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.763432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.764563 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.764564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.764564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.764573 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.766510 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767055 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.767056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.767057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.767116 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767320 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.767321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.767321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.767451 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.767497 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.767700 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.767778 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:22:53.768240 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.760591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33058)
0808 02:22:53.760782 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.760794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.760912 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.761110 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.761111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.761392 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.761510 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.761512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.762459 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.762460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.763020 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.763021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.763461 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.763462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.763531 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.764039 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.764045 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.764439 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.766613 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.766615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.766615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.766958 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.767190 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.767190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.767249 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.767548 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.767549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.767590 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.768030 (+ 440us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.768133 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":2,"spinlock_wait_cycles":10496,"thread_start_us":110,"threads_started":1}
I20260808 02:22:53.768514 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.760643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.30.160.61:38335)
0808 02:22:53.760720 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.760722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.760885 (+ 163us) server_negotiation.cc:407] Connection header received
0808 02:22:53.760934 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.760935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.760944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.761009 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.762486 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.763234 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.764183 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.764408 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.764493 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.765512 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.765513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.765514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.765520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.767264 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767611 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.767612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.767612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.767662 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767868 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.767869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.767869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.768030 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.768090 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.768245 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.768402 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":61}
I20260808 02:22:53.768530 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.760591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33604)
0808 02:22:53.760782 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.760794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.760871 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.761142 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.761142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.761393 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.761510 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.761512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.763320 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.763320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.764083 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.764086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.764496 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.764497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.764561 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.765003 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.765008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.765423 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.767374 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.767375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.767541 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.767737 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.767737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.767738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.767792 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.768110 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.768111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.768142 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.768216 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.768341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":19,"thread_start_us":108,"threads_started":1}
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.62@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
I20260808 02:22:53.778298 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.770709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33612)
0808 02:22:53.770833 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.770879 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.770940 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.771150 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.771152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.771390 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.771473 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.771476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.772462 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.772463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.773199 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.773202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.773652 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.773653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.773721 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.774176 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.774181 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.775646 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.777305 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.777323 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.777325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.777515 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.777715 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.777715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.777716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.777767 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.778028 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.778034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.778074 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.778134 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.778219 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260808 02:22:53.778347 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.770779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.30.160.61:38335)
0808 02:22:53.770880 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.770882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.770925 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:22:53.771073 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.771074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.771084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.771148 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.771578 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.772343 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.773283 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.773577 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.773663 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.775731 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.775733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.775733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.775740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.777230 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.777589 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.777590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.777590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.777647 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.777828 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.777829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.777829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.777961 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.778022 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.778180 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.778256 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:22:53.778981 5165 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.160.61:38335: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa"
member_type: NON_VOTER
last_known_addr {
host: "127.30.160.61"
port: 38335
}
attrs {
promote: true
}
}
dest_uuid: "bbceee63e42341518b449bd002318f22"
cas_config_opid_index: -1
I20260808 02:22:53.779290 5165 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } }
I20260808 02:22:53.779778 5769 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f0f0d4a6c0144573955a055e9eb58caa (127.30.160.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } }
I20260808 02:22:53.780128 5712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:53.780987 5769 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.160.61:38335
I20260808 02:22:53.781067 5712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Refusing update from remote peer bbceee63e42341518b449bd002318f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 02:22:53.781078 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } }
I20260808 02:22:53.781137 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:53.781327 5666 master.cc:537] Master@127.30.160.61:38335 shutting down...
I20260808 02:22:53.781427 5768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:22:53.782128 5149 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 02:22:53.782528 5149 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 -> Peer f0f0d4a6c0144573955a055e9eb58caa (127.30.160.61:38335): Couldn't send request to peer f0f0d4a6c0144573955a055e9eb58caa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 02:22:53.784057 5666 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:53.784147 5666 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:53.784194 5666 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: stopping tablet replica
I20260808 02:22:53.797475 5666 master.cc:552] Master@127.30.160.61:38335 shutdown complete.
I20260808 02:22:53.797557 5666 master_runner.cc:305] Clearing existing system tablet
I20260808 02:22:53.797724 5666 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.797803 5666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.799320 5666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:53.799964 5773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.800125 5666 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.800303 5666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:53.800369 5666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
uuid: "f0f0d4a6c0144573955a055e9eb58caa"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "a0892b3d6f5d4ba1055263d5101d3f47"
server_key_iv: "66486e5b460d5c28e21ddd84441e682f"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.801955 5666 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:53.803494 5666 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:22:53.803566 5666 log.cc:1196] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 02:22:53.803781 5666 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Deleting consensus metadata
I20260808 02:22:53.803961 5666 master_runner.cc:315] Copying system tablet from 127.30.160.62:36979
I20260808 02:22:53.804633 5666 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: tablet copy: Beginning tablet copy session from remote peer at address 127.30.160.62:36979
I20260808 02:22:53.812227 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.804942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33074)
0808 02:22:53.805150 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:53.805160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:53.805222 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:53.805369 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:53.805370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:53.805542 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:53.805594 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.805596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.806371 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.806372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.807173 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:53.807177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.807685 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.807686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:53.807761 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.808348 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:53.808356 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:53.808810 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:53.810932 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.810933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.810934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.811178 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.811427 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:53.811428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:53.811428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:53.811500 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:53.811827 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:53.811828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:53.811920 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 02:22:53.812011 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:22:53.812118 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":169,"threads_started":1}
I20260808 02:22:53.812255 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.804966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.30.160.62:36979)
0808 02:22:53.805123 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.805125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.805225 (+ 100us) server_negotiation.cc:407] Connection header received
0808 02:22:53.805286 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.805287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.805297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.805367 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.805673 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.806272 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.807281 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.807543 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.807597 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.808966 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.808968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.808968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.808975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.810828 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.811279 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.811280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.811281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.811347 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.811573 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.811575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.811575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.811738 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.811818 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.812053 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.812148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260808 02:22:53.813934 5187 tablet_copy_service.cc:140] P bbceee63e42341518b449bd002318f22: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0f0d4a6c0144573955a055e9eb58caa ({username='oryx', principal='oryx/127.30.160.61@KRBTEST.COM'} at 127.0.0.1:33074)
I20260808 02:22:53.814044 5187 tablet_copy_service.cc:161] P bbceee63e42341518b449bd002318f22: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0f0d4a6c0144573955a055e9eb58caa at {username='oryx', principal='oryx/127.30.160.61@KRBTEST.COM'} at 127.0.0.1:33074: session id = f0f0d4a6c0144573955a055e9eb58caa-00000000000000000000000000000000
I20260808 02:22:53.814848 5187 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:22:53.815528 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:22:53.816781 5666 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: tablet copy: Starting download of 0 data blocks...
I20260808 02:22:53.816943 5666 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: tablet copy: Starting download of 1 WAL segments...
I20260808 02:22:53.818086 5666 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:22:53.818988 5187 tablet_copy_service.cc:342] P bbceee63e42341518b449bd002318f22: Request end of tablet copy session f0f0d4a6c0144573955a055e9eb58caa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.160.61@KRBTEST.COM'} at 127.0.0.1:33074
I20260808 02:22:53.819049 5187 tablet_copy_service.cc:433] P bbceee63e42341518b449bd002318f22: ending tablet copy session f0f0d4a6c0144573955a055e9eb58caa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0f0d4a6c0144573955a055e9eb58caa
I20260808 02:22:53.822170 5666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:53.823486 5781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:53.823657 5783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.823709 5666 server_base.cc:1034] running on GCE node
W20260808 02:22:53.828058 5780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:53.828240 5666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:53.828419 5666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:53.829560 5666 hybrid_clock.cc:648] HybridClock initialized: now 1786155773829550 us; error 26 us; skew 500 ppm
Aug 08 02:22:53 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:53.831002 5666 init.cc:377] Logged in from keytab as oryx/127.30.160.61@KRBTEST.COM (short username oryx)
I20260808 02:22:53.831312 5666 webserver.cc:460] Webserver started at http://127.30.160.61:37799/ using document root <none> and password file <none>
I20260808 02:22:53.831435 5666 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:53.831492 5666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:53.832228 5666 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:53.832795 5790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:53.832957 5666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:22:53.833016 5666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
uuid: "f0f0d4a6c0144573955a055e9eb58caa"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "a0892b3d6f5d4ba1055263d5101d3f47"
server_key_iv: "66486e5b460d5c28e21ddd84441e682f"
server_key_version: "encryptionkey@0"
I20260808 02:22:53.833083 5666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:53.848369 5666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:53.850145 5666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:53.853065 5666 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.61:38335
I20260808 02:22:53.853482 5666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
I20260808 02:22:53.853503 5846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.61:38335 every 8 connection(s)
I20260808 02:22:53.854377 5847 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:22:53.854907 5847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Bootstrap starting.
I20260808 02:22:53.857686 5847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:22:53.857932 5847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Bootstrap complete.
I20260808 02:22:53.858129 5847 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } }
I20260808 02:22:53.858219 5847 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0f0d4a6c0144573955a055e9eb58caa, State: Initialized, Role: LEARNER
I20260808 02:22:53.858255 5847 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } }
I20260808 02:22:53.858417 5847 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:53.858413 5850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } }
I20260808 02:22:53.858538 5850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: LEARNER
I20260808 02:22:53.860031 5862 catalog_manager.cc:1259] Loaded cluster ID: 15468bf169d34060b08eb4005209c309
I20260808 02:22:53.860342 5862 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: loading cluster ID for follower catalog manager: success
I20260808 02:22:53.861479 5862 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: acquiring CA information for follower catalog manager: success
I20260808 02:22:53.861749 5862 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:22:53.905887 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:53.899227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.30.160.61:38335)
0808 02:22:53.899389 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:53.899391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:53.899400 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:53.899431 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:53.899431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:53.899440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:53.899516 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:53.900140 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.900852 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.901583 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:53.901803 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:53.901885 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:53.903139 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:53.903140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:53.903141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:53.903147 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:53.904664 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.905052 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.905053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.905054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.905128 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:53.905352 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:53.905353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:53.905353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:53.905466 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:53.905568 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:53.905683 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:22:53.905765 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.30.160.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.60.keytab.
Entry for principal oryx/127.30.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.60.keytab.
I20260808 02:22:53.931139 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:35609
--webserver_interface=127.30.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.60.keytab
--principal=oryx/127.30.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:54.014468 5866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:54.014693 5866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:54.014732 5866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:54.015858 5866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:54.015952 5866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:54.015990 5866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:54.016017 5866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:54.016039 5866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:54.017571 5866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.60.keytab
--principal=oryx/127.30.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:54.017840 5866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:54.018064 5866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:54.022806 5871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.022823 5874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.022840 5872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.023510 5866 server_base.cc:1034] running on GCE node
I20260808 02:22:54.023774 5866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:54.024081 5866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:54.025226 5866 hybrid_clock.cc:648] HybridClock initialized: now 1786155774025206 us; error 35 us; skew 500 ppm
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:54.027634 5866 init.cc:377] Logged in from keytab as oryx/127.30.160.60@KRBTEST.COM (short username oryx)
I20260808 02:22:54.028157 5866 webserver.cc:460] Webserver started at http://127.30.160.60:37427/ using document root <none> and password file <none>
I20260808 02:22:54.028384 5866 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:54.028452 5866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:54.028609 5866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:54.029629 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/instance:
uuid: "3a2e5751a1684453bbc957d4f3e85290"
format_stamp: "Formatted at 2026-08-08 02:22:54 on dist-test-slave-zkpd"
server_key: "417679b321cffe71e360ca40fd23a380"
server_key_iv: "0684ecc9b0d3f3d2cfd53ebd8293dd97"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.030011 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal/instance:
uuid: "3a2e5751a1684453bbc957d4f3e85290"
format_stamp: "Formatted at 2026-08-08 02:22:54 on dist-test-slave-zkpd"
server_key: "417679b321cffe71e360ca40fd23a380"
server_key_iv: "0684ecc9b0d3f3d2cfd53ebd8293dd97"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.031364 5866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:54.032191 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.032358 5866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:54.032465 5866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
uuid: "3a2e5751a1684453bbc957d4f3e85290"
format_stamp: "Formatted at 2026-08-08 02:22:54 on dist-test-slave-zkpd"
server_key: "417679b321cffe71e360ca40fd23a380"
server_key_iv: "0684ecc9b0d3f3d2cfd53ebd8293dd97"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.032539 5866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:54.058348 5866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:54.060652 5866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:54.060802 5866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:54.065641 5866 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.60:35609
I20260808 02:22:54.065696 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.60:35609 every 8 connection(s)
I20260808 02:22:54.066030 5866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
I20260808 02:22:54.066248 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5866
I20260808 02:22:54.067165 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:54.069108 5937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.160.60@KRBTEST.COM
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.60@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:54.076858 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.066880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.30.160.60:35609)
0808 02:22:54.067058 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.067066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.067089 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:22:54.067563 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.067569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.067616 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.067731 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.068534 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.069468 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.070423 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.070897 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.071051 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.073557 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.073564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.073564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.073571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.075229 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.075662 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.075664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.075665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.075777 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.076030 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.076031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.076032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.076263 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.076369 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.076520 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.076609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260808 02:22:54.077880 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.069738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33086)
0808 02:22:54.069885 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.069922 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.069986 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.070288 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.070290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.070545 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.070660 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.070663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.071620 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.071621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.072292 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.072295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.072714 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.072715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.072784 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.073256 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.073264 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.074829 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.076735 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.076742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.076744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.076972 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.077181 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.077181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.077182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.077267 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.077566 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.077571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.077623 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.077708 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.077811 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260808 02:22:54.077888 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.070061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.30.160.62:36979)
0808 02:22:54.070141 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.070144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.070151 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:54.070174 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.070175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.070184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.070250 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.070739 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.071368 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.072391 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.072598 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.072682 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.074922 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.074923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.074924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.074931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.076657 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.077037 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.077039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.077040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.077108 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.077341 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.077342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.077342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.077497 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.077562 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.077751 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.077828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:22:54.078454 5937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.60@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
I20260808 02:22:54.084393 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.078704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33634)
0808 02:22:54.078752 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.078762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.078797 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.079024 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.079025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.079144 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.079187 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.079188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.079793 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.079794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.080430 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.080432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.080749 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.080750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.080802 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.081152 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.081156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.082256 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.083580 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.083581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.083581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.083730 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.083952 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.083952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.083952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.084001 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.084191 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.084191 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.084222 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.084272 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.084336 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:22:54.084417 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.078793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.30.160.61:38335)
0808 02:22:54.078891 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.078893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.078902 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:54.078932 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.078933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.078947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.078996 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.079252 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.079721 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.080498 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.080686 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.080740 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.082332 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.082334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.082334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.082340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.083511 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.083793 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.083794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.083794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.083851 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.084048 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.084049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.084049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.084140 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.084187 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.084287 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.084350 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:22:54.084844 5937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.60@KRBTEST.COM for oryx/127.30.160.60@KRBTEST.COM
I20260808 02:22:54.090610 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.085093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56616)
0808 02:22:54.085141 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.085151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.085191 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.085310 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.085310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.085432 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.085454 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.085455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.086071 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.086072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.086523 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.086524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.086906 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.086907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.086955 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.087251 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.087254 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.088329 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.089626 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.089627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.089627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.089784 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.089994 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.089995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.089995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.090050 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.090293 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.090294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.090330 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.090414 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.090504 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:22:54.090665 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.085138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.30.160.60:35609)
0808 02:22:54.085170 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.085172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.085179 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:54.085246 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.085247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.085255 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.085306 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.085527 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.085970 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.086615 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.086822 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.086905 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.088424 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.088426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.088426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.088432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.089531 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.089872 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.089873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.089873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.089928 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.090120 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.090121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.090122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.090221 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.090290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.090435 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.090553 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":18}
I20260808 02:22:54.092103 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Bootstrap starting.
I20260808 02:22:54.092923 5937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:54.093228 5937 log.cc:824] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:54.094040 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: No bootstrap required, opened a new log
I20260808 02:22:54.095448 5937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } }
I20260808 02:22:54.095578 5937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:54.095602 5937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2e5751a1684453bbc957d4f3e85290, State: Initialized, Role: FOLLOWER
I20260808 02:22:54.095748 5937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } }
I20260808 02:22:54.096098 5937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:54.096071 5943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } } }
I20260808 02:22:54.096177 5943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:22:54.097934 5954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:54.098021 5954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:54.100353 5866 master_runner.cc:428] Detected that this master 3a2e5751a1684453bbc957d4f3e85290 is joining an existing cluster
I20260808 02:22:54.100419 5866 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.160.60:35609
I20260808 02:22:54.108181 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.101695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.30.160.60:35609)
0808 02:22:54.101729 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.101731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.102083 (+ 352us) server_negotiation.cc:407] Connection header received
0808 02:22:54.102168 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.102169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.102177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.102203 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.102587 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.103293 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104338 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.104610 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104650 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.105706 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.105707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.105707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.105713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.107089 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.107490 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.107491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.107491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.107538 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.107718 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.107719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.107719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.107802 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.107848 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.108061 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.108125 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 02:22:54.108346 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.101788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56620)
0808 02:22:54.102009 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.102018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.102105 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.102254 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.102255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.102442 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.102504 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.102506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.103383 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.103384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.104245 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.104248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.104732 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.104786 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.105173 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.105178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.105609 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.107174 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.107175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.107175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.107402 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.107592 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.107593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.107593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.107660 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.107866 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.107867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.107959 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.108041 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.108197 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":52,"thread_start_us":139,"threads_started":1}
I20260808 02:22:54.128412 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5231
I20260808 02:22:54.128763 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.101603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.30.160.61:38335)
0808 02:22:54.101643 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.101645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.101710 (+ 65us) server_negotiation.cc:407] Connection header received
0808 02:22:54.101804 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.101805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.101815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.101845 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.102416 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.103123 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104355 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.104643 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104684 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.125478 (+ 20794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.125480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.125480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.125490 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.127150 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.127595 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.127596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.127597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.127679 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.128057 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.128058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.128059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.128233 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.128286 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.128529 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.128639 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:54.129065 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.101429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.30.160.62:36979)
0808 02:22:54.101476 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.101478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.101635 (+ 157us) server_negotiation.cc:407] Connection header received
0808 02:22:54.101747 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.101748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.101759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.101792 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.102331 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.103110 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104347 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.104643 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104684 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.125813 (+ 21129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.125816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.125816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.125826 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.127597 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.128066 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.128068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.128068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.128143 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.128373 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.128374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.128374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.128529 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.128586 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.128859 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.128954 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:54.129123 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.101420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33642)
0808 02:22:54.101657 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.101666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.101730 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.101909 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.101910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.102267 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.102333 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.102334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.103213 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.103214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.104242 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.104245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.104764 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.104845 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.124746 (+ 19901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.124750 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.125355 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.127270 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.127271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.127272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.127503 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.127839 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.127841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.127841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.127954 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.128342 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.128343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.128397 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.128861 (+ 464us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.128992 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":74,"thread_start_us":91,"threads_started":1}
I20260808 02:22:54.129496 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.101395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33100)
0808 02:22:54.101565 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.101576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.101665 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.101874 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.101875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.102141 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.102223 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.102226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.103213 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.103214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.104242 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.104245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.104762 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.104763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.104872 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.124649 (+ 19777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.124653 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.125355 (+ 702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.127704 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.127705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.127706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.127973 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.128225 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.128226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.128226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.128292 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.128621 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.128622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.128669 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.129196 (+ 527us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.129329 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":26,"thread_start_us":88,"threads_started":1}
I20260808 02:22:54.140687 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:39219
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:36979
--webserver_port=35417
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155773, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.62@KRBTEST.COM for oryx/127.30.160.60@KRBTEST.COM
I20260808 02:22:54.149678 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.141028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56634)
0808 02:22:54.141118 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.141128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.141166 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.141469 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.141470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.141646 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.141691 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.141692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.142380 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.142381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.143246 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.143250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.143800 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.143802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.143944 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.144419 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.144426 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.146225 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.148375 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.148377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.148377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.148640 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.148891 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.148892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.148893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.148970 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.149295 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.149296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.149352 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.149439 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.149558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 02:22:54.149705 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.141263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.30.160.60:35609)
0808 02:22:54.141324 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.141326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.141334 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:54.141383 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.141384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.141392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.141460 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.141774 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.142277 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.143348 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.143694 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.143792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.146345 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.146347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.146347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.146356 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.148271 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.148742 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.148743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.148744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.148809 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.149045 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.149047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.149047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.149213 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.149292 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.149481 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.149571 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:22:54.150561 5165 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.160.60:35609: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290"
member_type: NON_VOTER
last_known_addr {
host: "127.30.160.60"
port: 35609
}
attrs {
promote: true
}
}
dest_uuid: "bbceee63e42341518b449bd002318f22"
cas_config_opid_index: 5
I20260808 02:22:54.150862 5165 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } }
I20260808 02:22:54.151412 5768 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.152621 5768 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.160.60:35609
I20260808 02:22:54.152711 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.152814 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.152983 5866 master.cc:537] Master@127.30.160.60:35609 shutting down...
W20260808 02:22:54.153488 5150 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 -> Peer 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60:35609): Couldn't send request to peer 3a2e5751a1684453bbc957d4f3e85290. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 02:22:54.156111 5866 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:54.156208 5866 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:54.156251 5866 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: stopping tablet replica
I20260808 02:22:54.157650 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.151556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33646)
0808 02:22:54.152357 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.152368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.152410 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.153627 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.153628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.153630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.153674 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.153676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.154735 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.154736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.156101 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.156104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.157367 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.157368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.157444 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.157468 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.157529 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.157571 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781}
I20260808 02:22:54.157961 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.152156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.30.160.61:38335)
0808 02:22:54.153397 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.153400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.153410 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:54.153439 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.153440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.153469 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.153542 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.153788 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.154613 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.156212 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.156918 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.157236 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.157328 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.157821 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.157893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1212}
I20260808 02:22:54.158174 5819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 LEARNER]: Refusing update from remote peer bbceee63e42341518b449bd002318f22: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:22:54.158545 5769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 02:22:54.159308 5819 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.160804 5850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbceee63e42341518b449bd002318f22. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.160897 5850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: LEARNER
I20260808 02:22:54.160933 5850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: NON_VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: true } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.160977 5850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: LEARNER
I20260808 02:22:54.161001 5769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: attempting to promote NON_VOTER f0f0d4a6c0144573955a055e9eb58caa to VOTER
I20260808 02:22:54.161175 5769 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } }
I20260808 02:22:54.161585 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.161474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56646)
0808 02:22:54.161517 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.161557 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 02:22:54.161834 5819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 LEARNER]: Refusing update from remote peer bbceee63e42341518b449bd002318f22: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:22:54.161859 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.161790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56656)
0808 02:22:54.161820 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.161838 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260808 02:22:54.161911 5150 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 -> Peer 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60:35609): Couldn't send request to peer 3a2e5751a1684453bbc957d4f3e85290. Status: Network error: Client connection negotiation failed: client connection to 127.30.160.60:35609: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:22:54.162117 5965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 02:22:54.162725 5768 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0f0d4a6c0144573955a055e9eb58caa (127.30.160.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.162879 5819 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0f0d4a6c0144573955a055e9eb58caa (127.30.160.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.163583 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.163655 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.163704 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.163745 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.164037 5968 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.164134 5968 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:54.169869 5866 master.cc:552] Master@127.30.160.60:35609 shutdown complete.
I20260808 02:22:54.169935 5866 master_runner.cc:305] Clearing existing system tablet
I20260808 02:22:54.170066 5866 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:54.170106 5866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:54.170933 5866 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:54.171502 5970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.171648 5866 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:54.171696 5866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.171718 5866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
uuid: "3a2e5751a1684453bbc957d4f3e85290"
format_stamp: "Formatted at 2026-08-08 02:22:54 on dist-test-slave-zkpd"
server_key: "417679b321cffe71e360ca40fd23a380"
server_key_iv: "0684ecc9b0d3f3d2cfd53ebd8293dd97"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.172673 5866 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:54.174292 5866 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:22:54.174366 5866 log.cc:1196] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 02:22:54.174566 5866 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Deleting consensus metadata
I20260808 02:22:54.174685 5866 master_runner.cc:315] Copying system tablet from 127.30.160.62:36979
I20260808 02:22:54.175287 5866 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: tablet copy: Beginning tablet copy session from remote peer at address 127.30.160.62:36979
I20260808 02:22:54.182663 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.175585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33106)
0808 02:22:54.175749 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.175759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.175834 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.176002 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.176002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.176207 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.176276 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.176279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.177210 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.177211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.178031 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.178034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.178499 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.178500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.178575 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.179104 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.179109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.179515 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.181491 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.181492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.181492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.181716 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.181968 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.181969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.181969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.182042 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.182342 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.182343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.182391 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.182466 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.182571 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260808 02:22:54.182685 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.175655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.30.160.62:36979)
0808 02:22:54.175702 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.175705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.175836 (+ 131us) server_negotiation.cc:407] Connection header received
0808 02:22:54.175906 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.175907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.175920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.175987 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.176353 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.177116 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.178115 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.178411 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.178483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.179597 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.179598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.179598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.179605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.181392 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.181824 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.181826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.181826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.181891 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.182126 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.182127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.182127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.182267 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.182343 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.182514 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.182598 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:54.183305 5187 tablet_copy_service.cc:140] P bbceee63e42341518b449bd002318f22: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3a2e5751a1684453bbc957d4f3e85290 ({username='oryx', principal='oryx/127.30.160.60@KRBTEST.COM'} at 127.0.0.1:33106)
I20260808 02:22:54.183367 5187 tablet_copy_service.cc:161] P bbceee63e42341518b449bd002318f22: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3a2e5751a1684453bbc957d4f3e85290 at {username='oryx', principal='oryx/127.30.160.60@KRBTEST.COM'} at 127.0.0.1:33106: session id = 3a2e5751a1684453bbc957d4f3e85290-00000000000000000000000000000000
I20260808 02:22:54.184026 5187 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:22:54.184527 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:22:54.185904 5866 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: tablet copy: Starting download of 0 data blocks...
I20260808 02:22:54.186131 5866 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: tablet copy: Starting download of 1 WAL segments...
I20260808 02:22:54.186962 5866 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:22:54.187873 5187 tablet_copy_service.cc:342] P bbceee63e42341518b449bd002318f22: Request end of tablet copy session 3a2e5751a1684453bbc957d4f3e85290-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.160.60@KRBTEST.COM'} at 127.0.0.1:33106
I20260808 02:22:54.187969 5187 tablet_copy_service.cc:433] P bbceee63e42341518b449bd002318f22: ending tablet copy session 3a2e5751a1684453bbc957d4f3e85290-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3a2e5751a1684453bbc957d4f3e85290
I20260808 02:22:54.190433 5866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:54.191591 5980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.192657 5981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.192936 5983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.193306 5866 server_base.cc:1034] running on GCE node
I20260808 02:22:54.193405 5866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:54.193542 5866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:54.194680 5866 hybrid_clock.cc:648] HybridClock initialized: now 1786155774194671 us; error 24 us; skew 500 ppm
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:54.196395 5866 init.cc:377] Logged in from keytab as oryx/127.30.160.60@KRBTEST.COM (short username oryx)
I20260808 02:22:54.196666 5866 webserver.cc:460] Webserver started at http://127.30.160.60:44231/ using document root <none> and password file <none>
I20260808 02:22:54.196789 5866 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:54.196820 5866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:54.197546 5866 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:54.198127 5990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.198308 5866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.198407 5866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
uuid: "3a2e5751a1684453bbc957d4f3e85290"
format_stamp: "Formatted at 2026-08-08 02:22:54 on dist-test-slave-zkpd"
server_key: "417679b321cffe71e360ca40fd23a380"
server_key_iv: "0684ecc9b0d3f3d2cfd53ebd8293dd97"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.198536 5866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:54.209005 5866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:54.211040 5866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:54.214493 5866 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.60:35609
I20260808 02:22:54.214579 6046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.60:35609 every 8 connection(s)
I20260808 02:22:54.215034 5866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
I20260808 02:22:54.216040 6047 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:22:54.216568 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Bootstrap starting.
I20260808 02:22:54.219435 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:22:54.219671 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: Bootstrap complete.
I20260808 02:22:54.219868 6047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } }
I20260808 02:22:54.219977 6047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a2e5751a1684453bbc957d4f3e85290, State: Initialized, Role: LEARNER
I20260808 02:22:54.220014 6047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } }
I20260808 02:22:54.220187 6047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:54.220197 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.220458 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:22:54.222612 6061 catalog_manager.cc:1259] Loaded cluster ID: 15468bf169d34060b08eb4005209c309
I20260808 02:22:54.222664 6061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: loading cluster ID for follower catalog manager: success
I20260808 02:22:54.223765 6061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: acquiring CA information for follower catalog manager: success
I20260808 02:22:54.224100 6061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:22:54.244632 5964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:54.244832 5964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:54.244855 5964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:54.245995 5964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:54.246037 5964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:54.246063 5964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:54.247553 5964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:39219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=35417
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:54.247822 5964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:54.248070 5964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:54.253139 6068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.253175 6067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.253175 6070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.253453 5964 server_base.cc:1034] running on GCE node
I20260808 02:22:54.253659 5964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:54.253942 5964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:54.255110 5964 hybrid_clock.cc:648] HybridClock initialized: now 1786155774255086 us; error 39 us; skew 500 ppm
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:54.257946 5964 init.cc:377] Logged in from keytab as oryx/127.30.160.1@KRBTEST.COM (short username oryx)
I20260808 02:22:54.258482 5964 webserver.cc:460] Webserver started at http://127.30.160.1:35417/ using document root <none> and password file <none>
I20260808 02:22:54.258744 5964 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:54.258831 5964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:54.260746 5964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:54.261394 6077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.261596 5964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:54.261684 5964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "3177ad2f6fcb4ef0962396eb5ad844d3"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "69adf7c25f8bcaa2110c561ef5624032"
server_key_iv: "b9a554d6f855868a5d7c7795d5262b7e"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.262094 5964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:54.282199 5964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:54.284035 5964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:54.284149 5964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:54.284341 5964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:54.284662 5964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:54.284694 5964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.284750 5964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:54.284776 5964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.290659 5964 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:39219
I20260808 02:22:54.290704 6200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:39219 every 8 connection(s)
I20260808 02:22:54.291065 5964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:54.297434 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 5964
I20260808 02:22:54.298095 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5374
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.60@KRBTEST.COM
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:54.304795 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.291784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.30.160.1:57473)
0808 02:22:54.292445 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.292453 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.293154 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.293415 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.293416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.293931 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.294280 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.294282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.295533 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.295534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.296668 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.296671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.297213 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.297214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.297361 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.297893 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.297904 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.300906 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.302939 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.302947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.302949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.303274 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.303525 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.303526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.303526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.303680 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.304054 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.304063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.304231 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.304360 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.304542 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":15,"thread_start_us":93,"threads_started":1}
I20260808 02:22:54.305779 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.292090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:57473 (local address 127.30.160.60:35609)
0808 02:22:54.292545 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.292547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.292554 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:54.293309 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.293310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.293331 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.293401 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.294391 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.295409 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.296769 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.297096 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.297151 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.301014 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.301016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.301016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.301024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.302839 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.303381 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.303383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.303383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.303447 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.303764 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.303765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.303766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.303946 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.304026 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.304434 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.304869 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":58,"threads_started":1}
I20260808 02:22:54.306712 6201 heartbeater.cc:347] Connected to a master server at 127.30.160.60:35609
I20260808 02:22:54.306759 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:34901
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:36979
--webserver_port=37361
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:22:54.306808 6201 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.306999 6201 heartbeater.cc:511] Master 127.30.160.60:35609 requested a full tablet report, sending...
I20260808 02:22:54.307335 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.291981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:50889 (local address 127.30.160.62:36979)
0808 02:22:54.292079 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.292081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.292321 (+ 240us) server_negotiation.cc:407] Connection header received
0808 02:22:54.293248 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.293249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.293263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.293332 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.294386 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.295154 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.296778 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.297084 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.297129 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.303767 (+ 6638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.303768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.303768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.303776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.305688 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.306357 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.306358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.306359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.306442 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.306742 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.306743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.306744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.306893 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.306952 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.307155 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.307243 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:22:54.307503 6009 ts_manager.cc:194] Registered new tserver with Master: 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219)
I20260808 02:22:54.307601 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.291782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:50889)
0808 02:22:54.292178 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.292225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.293132 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.293407 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.293411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.294041 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.294261 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.294269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.295246 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.295247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.296668 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.296670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.297214 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.297215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.301681 (+ 4466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.302416 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.302420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.303669 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.306056 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.306057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.306057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.306257 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.306523 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.306524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.306524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.306661 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.306987 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.306988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.307026 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.307410 (+ 384us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.307508 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":86,"spinlock_wait_cycles":6016,"thread_start_us":195,"threads_started":1}
I20260808 02:22:54.308606 6202 heartbeater.cc:347] Connected to a master server at 127.30.160.62:36979
I20260808 02:22:54.308655 6202 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.308724 6202 heartbeater.cc:511] Master 127.30.160.62:36979 requested a full tablet report, sending...
I20260808 02:22:54.308977 5165 ts_manager.cc:194] Re-registered known tserver with Master: 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219)
I20260808 02:22:54.309484 5165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:50889
I20260808 02:22:54.311496 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.292087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:45469 (local address 127.30.160.61:38335)
0808 02:22:54.292184 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.292187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.292383 (+ 196us) server_negotiation.cc:407] Connection header received
0808 02:22:54.293264 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.293266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.293279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.293311 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.294375 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.295066 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.296786 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.297129 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.297179 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.306916 (+ 9737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.306918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.306918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.306926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.310379 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.310703 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.310704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.310705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.310765 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.310971 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.310972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.310972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.311092 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.311162 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.311321 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.311406 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:22:54.311512 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.291783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.30.160.1:45469)
0808 02:22:54.292326 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.292336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.293154 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.293407 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.293412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.293939 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.294260 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.294266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.295215 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.295216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.296670 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.296674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.297269 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.297270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.297366 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.298064 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.298070 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.306786 (+ 8716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.310453 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.310454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.310454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.310612 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.310840 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.310840 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.310841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.310900 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.311168 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.311168 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.311216 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.311283 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.311407 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":58,"thread_start_us":327,"threads_started":1}
I20260808 02:22:54.311930 6203 heartbeater.cc:347] Connected to a master server at 127.30.160.61:38335
I20260808 02:22:54.312005 6203 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.312079 6203 heartbeater.cc:511] Master 127.30.160.61:38335 requested a full tablet report, sending...
I20260808 02:22:54.312441 5808 ts_manager.cc:194] Registered new tserver with Master: 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219)
W20260808 02:22:54.391441 6209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:54.391633 6209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:54.391660 6209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:54.392889 6209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:54.392932 6209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:54.392959 6209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:54.394435 6209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:34901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=37361
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:54.394673 6209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:54.394910 6209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:54.399852 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.399923 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.399925 6209 server_base.cc:1034] running on GCE node
W20260808 02:22:54.399860 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.400276 6209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:54.400560 6209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:54.401715 6209 hybrid_clock.cc:648] HybridClock initialized: now 1786155774401710 us; error 36 us; skew 500 ppm
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:54.404109 6209 init.cc:377] Logged in from keytab as oryx/127.30.160.2@KRBTEST.COM (short username oryx)
I20260808 02:22:54.404582 6209 webserver.cc:460] Webserver started at http://127.30.160.2:37361/ using document root <none> and password file <none>
I20260808 02:22:54.404809 6209 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:54.404891 6209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:54.406237 6209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:54.406800 6224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.406983 6209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.407083 6209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "1cf3c56e80e94739a449813296fb5f2e"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "7e329705cb209cf12e3e802aeef3cee4"
server_key_iv: "09a5ef7b20ab903a3f78d5fea8b7bb8a"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.407429 6209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:54.420208 6209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:54.422122 6209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:54.422276 6209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:54.422497 6209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:54.422864 6209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:54.422930 6209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.422971 6209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:54.423000 6209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.429059 6209 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:34901
I20260808 02:22:54.429106 6343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:34901 every 8 connection(s)
I20260808 02:22:54.429430 6209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:54.432885 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 6209
I20260808 02:22:54.433061 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5521
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.60@KRBTEST.COM
I20260808 02:22:54.441694 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.430359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:59855 (local address 127.30.160.60:35609)
0808 02:22:54.430406 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.430408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.430493 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:22:54.431489 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.431491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.431504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.431537 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.432386 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.433184 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.435275 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.435707 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.435779 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.438067 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.438068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.438069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.438077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.440064 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.440439 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.440440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.440440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.440506 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.440830 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.440831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.440832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.440996 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.441039 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.441429 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.441549 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:54.442241 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.430076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.30.160.2:59855)
0808 02:22:54.430431 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.430440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.431334 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.431627 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.431631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.432064 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.432268 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.432274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.433562 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.433562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.434622 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.434625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.435852 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.435853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.435965 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.436397 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.436403 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.437975 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.440151 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.440152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.440153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.440368 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.440593 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.440594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.440594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.440752 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.441088 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.441095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.441252 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.441788 (+ 536us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.441973 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":21,"thread_start_us":93,"threads_started":1}
I20260808 02:22:54.442673 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.430454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:34599 (local address 127.30.160.62:36979)
0808 02:22:54.430506 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.430509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.430520 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:54.431490 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.431491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.431504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.431537 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.432389 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.433164 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.434721 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.435115 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.435207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.437610 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.437612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.437612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.437621 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.439451 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.439846 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.439848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.439848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.439937 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.441879 (+ 1942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.441881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.441881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.442025 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.442070 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.442539 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.442608 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:22:54.442989 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.430079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:34599)
0808 02:22:54.430253 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.430298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.431326 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.431627 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.431631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.432062 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.432268 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.432274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.433307 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.433307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.434622 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.434625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.435208 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.435209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.435306 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.435814 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.435824 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.437487 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.439530 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.439538 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.439540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.439761 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.441633 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.441634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.441634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.441702 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.442116 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.442117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.442157 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.442816 (+ 659us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.442919 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":26,"thread_start_us":98,"threads_started":1}
I20260808 02:22:54.443423 6345 heartbeater.cc:347] Connected to a master server at 127.30.160.62:36979
I20260808 02:22:54.443532 6345 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.443704 6345 heartbeater.cc:511] Master 127.30.160.62:36979 requested a full tablet report, sending...
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
I20260808 02:22:54.444136 5165 ts_manager.cc:194] Re-registered known tserver with Master: 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2:34901)
I20260808 02:22:54.445240 5165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:34599
I20260808 02:22:54.446911 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.430405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:39199 (local address 127.30.160.61:38335)
0808 02:22:54.430514 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.430516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.430601 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:22:54.431453 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.431455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.431483 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.431543 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.432366 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.433093 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.441619 (+ 8526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.441950 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.442011 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.444158 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.444160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.444160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.444169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.445735 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.446133 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.446135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.446135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.446195 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.446427 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.446428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.446428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.446566 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.446634 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.446790 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.446850 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:22:54.447242 6344 heartbeater.cc:347] Connected to a master server at 127.30.160.60:35609
I20260808 02:22:54.447304 6344 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.447381 6344 heartbeater.cc:511] Master 127.30.160.60:35609 requested a full tablet report, sending...
I20260808 02:22:54.447453 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.430076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.30.160.2:39199)
0808 02:22:54.430537 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.430545 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.431325 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.431632 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.431633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.432058 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.432266 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.432271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.440984 (+ 8713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.440984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.441517 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.441518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.442068 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.442069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.442140 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.442531 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.442536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.444076 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.445811 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.445812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.445813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.446042 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.446268 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.446269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.446270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.446347 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.446643 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.446644 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.446691 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.446766 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.446890 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":28,"spinlock_wait_cycles":6400,"thread_start_us":92,"threads_started":1}
I20260808 02:22:54.447732 6009 ts_manager.cc:194] Registered new tserver with Master: 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2:34901)
I20260808 02:22:54.447788 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:34837
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:36979
--webserver_port=43081
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:22:54.448078 6346 heartbeater.cc:347] Connected to a master server at 127.30.160.61:38335
I20260808 02:22:54.448156 6346 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.448203 6346 heartbeater.cc:511] Master 127.30.160.61:38335 requested a full tablet report, sending...
I20260808 02:22:54.448501 5808 ts_manager.cc:194] Registered new tserver with Master: 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2:34901)
W20260808 02:22:54.531000 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:54.531194 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:54.531219 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:54.532384 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:54.532424 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:54.532450 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:22:54.534046 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:34837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=43081
--tserver_master_addrs=127.30.160.62:36979,127.30.160.61:38335,127.30.160.60:35609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:54.534291 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:54.534525 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:54.539397 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:54.539448 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.539534 6351 server_base.cc:1034] running on GCE node
W20260808 02:22:54.539389 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:54.539810 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:54.540076 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:54.541235 6351 hybrid_clock.cc:648] HybridClock initialized: now 1786155774541225 us; error 40 us; skew 500 ppm
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:54.543849 6351 init.cc:377] Logged in from keytab as oryx/127.30.160.3@KRBTEST.COM (short username oryx)
I20260808 02:22:54.544350 6351 webserver.cc:460] Webserver started at http://127.30.160.3:43081/ using document root <none> and password file <none>
I20260808 02:22:54.544600 6351 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:54.544657 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:54.546002 6351 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:54.546659 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.546851 6351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:54.546947 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "b0c55e028888437e9969d643eff2f5de"
format_stamp: "Formatted at 2026-08-08 02:22:53 on dist-test-slave-zkpd"
server_key: "b2588fd30b61899b033bbfb6562068e9"
server_key_iv: "d404ef14b9c71a5757662ff233e9e979"
server_key_version: "encryptionkey@0"
I20260808 02:22:54.547272 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:54.559688 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:54.561618 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:54.561728 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:54.561905 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:22:54.562219 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:54.562256 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.562300 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:54.562326 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:54.568265 6351 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:34837
I20260808 02:22:54.568331 6482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:34837 every 8 connection(s)
I20260808 02:22:54.568645 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:22:54.573168 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 6351
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.61@KRBTEST.COM
Aug 08 02:22:54 dist-test-slave-zkpd krb5kdc[5118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155774, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.60@KRBTEST.COM
I20260808 02:22:54.581880 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.569432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:40659 (local address 127.30.160.60:35609)
0808 02:22:54.569506 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.569508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.569989 (+ 481us) server_negotiation.cc:407] Connection header received
0808 02:22:54.571137 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.571138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.571150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.571207 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.572128 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.572627 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574522 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.574775 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574850 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.578922 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.578924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.578924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.578932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.580589 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.581022 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.581023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.581024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.581107 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.581330 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.581331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.581331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.581459 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.581500 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.581711 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.581793 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:22:54.582034 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.569392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.30.160.3:40659)
0808 02:22:54.569896 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.569905 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.571020 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.571292 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.571298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.571738 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.572027 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.572032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.572735 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.572735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.574128 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.574132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.574864 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.574930 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.575713 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.575717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.578821 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.580711 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.580712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.580713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.580932 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.581185 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.581186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.581186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.581239 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.581552 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.581553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.581598 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.581683 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.581835 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":83,"thread_start_us":167,"threads_started":1}
I20260808 02:22:54.582561 6483 heartbeater.cc:347] Connected to a master server at 127.30.160.60:35609
I20260808 02:22:54.582599 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.569503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:55141 (local address 127.30.160.62:36979)
0808 02:22:54.569616 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.569618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.570118 (+ 500us) server_negotiation.cc:407] Connection header received
0808 02:22:54.571144 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.571146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.571159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.571193 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.572144 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.572896 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574206 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.574422 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574472 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.578216 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.578217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.578218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.578225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.580286 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.580739 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.580740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.580740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.580802 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.581089 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.581090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.581090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.581222 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.581281 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.582406 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.582486 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:22:54.582659 6483 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.582850 6483 heartbeater.cc:511] Master 127.30.160.60:35609 requested a full tablet report, sending...
I20260808 02:22:54.582943 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.569474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:55141)
0808 02:22:54.570044 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.570052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.571026 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.571292 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.571296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.571741 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.572029 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.572034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.572997 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.572997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.574128 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.574131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.574674 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.574816 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.575449 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.575458 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.578126 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.580372 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.580379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.580381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.580651 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.580885 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.580886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.580886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.581006 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.581308 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.581316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.581482 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.581592 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.581782 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":49,"thread_start_us":109,"threads_started":1}
I20260808 02:22:54.583382 6009 ts_manager.cc:194] Registered new tserver with Master: b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837)
I20260808 02:22:54.583506 6484 heartbeater.cc:347] Connected to a master server at 127.30.160.62:36979
I20260808 02:22:54.583549 6484 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.583608 6484 heartbeater.cc:511] Master 127.30.160.62:36979 requested a full tablet report, sending...
I20260808 02:22:54.583859 5165 ts_manager.cc:194] Re-registered known tserver with Master: b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837)
I20260808 02:22:54.584573 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.575238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.30.160.60:35609)
0808 02:22:54.575689 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.575692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.575700 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:54.575721 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.575722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.575733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.575815 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.576764 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.577527 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.578613 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.578948 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.579023 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.581412 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.581413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.581413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.581419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.583119 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.583503 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.583504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.583504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.583563 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.583787 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.583788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.583788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.583944 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.584038 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.584403 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.584484 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":58,"threads_started":1}
I20260808 02:22:54.584933 5165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:55141
I20260808 02:22:54.587332 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.569562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:45565 (local address 127.30.160.61:38335)
0808 02:22:54.569649 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.569652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.569920 (+ 268us) server_negotiation.cc:407] Connection header received
0808 02:22:54.571144 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.571146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.571159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.571193 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.572143 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.572903 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574235 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.574530 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574575 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.583877 (+ 9302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.583879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.583879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.583886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.585948 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.586384 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.586385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.586385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.586444 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.586672 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.586673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.586674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.586837 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.586936 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.587128 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.587217 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:22:54.587352 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.569391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.30.160.3:45565)
0808 02:22:54.569774 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.569833 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.571014 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.571291 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.571296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.571745 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:54.572028 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.572034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.572997 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.572997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.574128 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.574131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.574672 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.574673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.574827 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.575524 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:54.575529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:54.583765 (+ 8236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:54.586048 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.586049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.586049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.586285 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.586521 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:54.586522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:54.586522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:54.586584 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:54.586960 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:54.586961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:54.587009 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.587090 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.587204 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":101,"thread_start_us":179,"threads_started":1}
I20260808 02:22:54.587690 6485 heartbeater.cc:347] Connected to a master server at 127.30.160.61:38335
I20260808 02:22:54.587750 6485 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:54.587819 6485 heartbeater.cc:511] Master 127.30.160.61:38335 requested a full tablet report, sending...
I20260808 02:22:54.588140 5808 ts_manager.cc:194] Registered new tserver with Master: b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837)
I20260808 02:22:54.589416 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.574562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.30.160.61:38335)
0808 02:22:54.574698 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.574700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.576418 (+ 1718us) server_negotiation.cc:407] Connection header received
0808 02:22:54.576440 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.576441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.576453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.576507 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.577456 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.579809 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.584506 (+ 4697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.584799 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.584864 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.586171 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.586173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.586173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.586180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.588152 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.588543 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.588545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.588545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.588602 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.588825 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.588826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.588826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.588961 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.589063 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.589209 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.589299 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260808 02:22:54.590024 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.575795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.30.160.62:36979)
0808 02:22:54.576216 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.576219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.576685 (+ 466us) server_negotiation.cc:407] Connection header received
0808 02:22:54.576709 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.576710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.576721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.576782 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.578117 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.578954 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.580843 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.581151 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.581207 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.586636 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.586638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.586638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.586645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.588862 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.589247 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.589248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.589248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.589312 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.589516 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.589516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.589517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.589623 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.589728 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.589837 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.589921 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":80,"threads_started":1}
I20260808 02:22:54.591008 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.587103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.30.160.60:35609)
0808 02:22:54.587141 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.587143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.587150 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:54.587170 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.587171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.587196 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.587939 (+ 743us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.587975 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.588767 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.590103 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.590648 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.590740 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.590827 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.590904 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.590947 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:54.591003 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.586896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56684)
0808 02:22:54.586978 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.586989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.587030 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.587284 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.587285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.587287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.587324 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.587326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.588890 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.588891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.589998 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.590002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.590769 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.590770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.590817 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.590838 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.590896 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.590937 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 02:22:54.591364 6020 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 02:22:54.591550 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbceee63e42341518b449bd002318f22. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.591658 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:22:54.591660 5165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:54.593261 5165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:54.593914 5965 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: NON_VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: true } } }
I20260808 02:22:54.594004 5965 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.595166 5769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: attempting to promote NON_VOTER 3a2e5751a1684453bbc957d4f3e85290 to VOTER
I20260808 02:22:54.595284 5769 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:54.595662 5819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Refusing update from remote peer bbceee63e42341518b449bd002318f22: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 02:22:54.595834 6020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 LEARNER]: Refusing update from remote peer bbceee63e42341518b449bd002318f22: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 02:22:54.595854 5965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 02:22:54.596407 5768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 02:22:54.596562 5768 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.596717 5819 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.597342 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.597427 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.597478 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.597534 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.597600 5968 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.597687 5968 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:54.598284 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.598371 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:54.598434 6020 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.599491 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbceee63e42341518b449bd002318f22" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:54.599594 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:54.603156 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.599233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.30.160.3:34837)
0808 02:22:54.600057 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.600063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.600073 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:54.600109 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.600111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.600126 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.600211 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.600369 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.601307 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602308 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.602786 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602884 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.602969 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.603040 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.603079 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":93,"threads_started":1}
I20260808 02:22:54.603243 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.598545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:34901 (local address 127.0.0.1:47606)
0808 02:22:54.598809 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.598820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.598864 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.599337 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.599338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.599340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.599384 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.599386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.601119 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.601120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.602110 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.602112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.602977 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.603040 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.603043 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.603077 (+ 34us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.603122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":80,"threads_started":1}
I20260808 02:22:54.603156 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.598623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34837 (local address 127.0.0.1:37092)
0808 02:22:54.598926 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.598937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.598983 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.600227 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.600228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.600231 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.600279 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.600281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.601398 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.601399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.602222 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.602225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.602898 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.602963 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.602968 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.603025 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.603067 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":67,"threads_started":1}
I20260808 02:22:54.603457 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.598982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.30.160.2:34901)
0808 02:22:54.599146 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.599151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.599163 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:22:54.599204 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.599206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.599224 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.599462 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.599530 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.601010 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602194 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.602883 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.603205 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.603298 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.603329 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.603380 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260808 02:22:54.603870 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.598520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39219 (local address 127.0.0.1:48288)
0808 02:22:54.598738 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.598749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.598812 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.599734 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.599735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.599737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.599780 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.599782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.600805 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.600806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.601956 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.601959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.603702 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.603703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.603750 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.603753 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.603790 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.603820 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260808 02:22:54.603973 6414 tablet_service.cc:1467] Processing CreateTablet for tablet 112f14f0b49d484b8570e101d33fb66b (DEFAULT_TABLE table=test-table [id=87a52bcfb1d24d42a7b3a80098b0be83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:54.604192 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.599267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.30.160.1:39219)
0808 02:22:54.599559 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.599564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.599573 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:54.599633 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.599636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.599653 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.599736 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.599839 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.600722 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602054 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.602803 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.602876 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.604004 (+ 1128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.604043 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.604093 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
I20260808 02:22:54.604359 6414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112f14f0b49d484b8570e101d33fb66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:54.604537 6276 tablet_service.cc:1467] Processing CreateTablet for tablet 112f14f0b49d484b8570e101d33fb66b (DEFAULT_TABLE table=test-table [id=87a52bcfb1d24d42a7b3a80098b0be83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:54.604915 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112f14f0b49d484b8570e101d33fb66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:54.608124 6513 tablet_bootstrap.cc:492] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Bootstrap starting.
I20260808 02:22:54.608386 6514 tablet_bootstrap.cc:492] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Bootstrap starting.
I20260808 02:22:54.608808 6513 tablet_bootstrap.cc:654] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:54.608734 6131 tablet_service.cc:1467] Processing CreateTablet for tablet 112f14f0b49d484b8570e101d33fb66b (DEFAULT_TABLE table=test-table [id=87a52bcfb1d24d42a7b3a80098b0be83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:54.609027 6514 tablet_bootstrap.cc:654] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:54.609129 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112f14f0b49d484b8570e101d33fb66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:54.609135 6513 log.cc:824] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:54.609391 6514 log.cc:824] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:54.610036 6513 tablet_bootstrap.cc:492] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: No bootstrap required, opened a new log
I20260808 02:22:54.610116 6513 ts_tablet_manager.cc:1397] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:22:54.610354 6514 tablet_bootstrap.cc:492] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: No bootstrap required, opened a new log
I20260808 02:22:54.610477 6514 ts_tablet_manager.cc:1397] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:54.611930 6517 tablet_bootstrap.cc:492] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Bootstrap starting.
I20260808 02:22:54.612226 6513 raft_consensus.cc:348] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.612439 6513 raft_consensus.cc:374] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:54.612483 6513 raft_consensus.cc:729] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c55e028888437e9969d643eff2f5de, State: Initialized, Role: FOLLOWER
I20260808 02:22:54.612617 6517 tablet_bootstrap.cc:654] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:54.612603 6513 consensus_queue.cc:260] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.612972 6517 log.cc:824] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:54.613261 6514 raft_consensus.cc:348] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.613421 6514 raft_consensus.cc:374] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:54.613456 6514 raft_consensus.cc:729] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf3c56e80e94739a449813296fb5f2e, State: Initialized, Role: FOLLOWER
I20260808 02:22:54.613575 6514 consensus_queue.cc:260] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.613755 6517 tablet_bootstrap.cc:492] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: No bootstrap required, opened a new log
I20260808 02:22:54.612905 6484 heartbeater.cc:503] Master 127.30.160.62:36979 was elected leader, sending a full tablet report...
I20260808 02:22:54.613829 6517 ts_tablet_manager.cc:1397] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:22:54.613874 6514 ts_tablet_manager.cc:1428] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:22:54.612852 6513 ts_tablet_manager.cc:1428] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:22:54.614306 6345 heartbeater.cc:503] Master 127.30.160.62:36979 was elected leader, sending a full tablet report...
I20260808 02:22:54.615854 6517 raft_consensus.cc:348] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.616075 6517 raft_consensus.cc:374] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:54.616128 6517 raft_consensus.cc:729] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3177ad2f6fcb4ef0962396eb5ad844d3, State: Initialized, Role: FOLLOWER
I20260808 02:22:54.616248 6517 consensus_queue.cc:260] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.616505 6517 ts_tablet_manager.cc:1428] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 02:22:54.616570 6202 heartbeater.cc:503] Master 127.30.160.62:36979 was elected leader, sending a full tablet report...
W20260808 02:22:54.680181 6347 tablet.cc:2369] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:22:54.792640 6204 tablet.cc:2369] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:22:54.820430 6486 tablet.cc:2369] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:54.851732 6520 raft_consensus.cc:482] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:54.851853 6520 raft_consensus.cc:504] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.852250 6520 leader_election.cc:288] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219), b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837)
I20260808 02:22:54.855554 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.852442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39219 (local address 127.30.160.2:39885)
0808 02:22:54.852512 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.852526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.852591 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.852748 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.852749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.852751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.852848 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.852850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.853534 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.853535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.854699 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.854701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.855305 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.855306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.855387 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.855391 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.855449 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.855499 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 02:22:54.855597 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.852530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:39885 (local address 127.30.160.1:39219)
0808 02:22:54.852577 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.852580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.852590 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:54.852689 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.852690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.852700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.852747 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.852931 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.853445 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.854801 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.855222 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.855288 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.855295 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.855464 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.855506 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:54.855726 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.852587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34837 (local address 127.30.160.2:55725)
0808 02:22:54.852633 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:54.852642 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:54.852679 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:54.852838 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:54.852839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:54.852841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:54.852944 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.852946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.853826 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.853827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.854847 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:54.854848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.855416 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.855417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:54.855520 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.855523 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:54.855584 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:22:54.855633 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 02:22:54.855741 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.852631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:55725 (local address 127.30.160.3:34837)
0808 02:22:54.852696 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.852698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.852729 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:22:54.852753 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.852754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.852762 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.852832 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:54.853038 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.853727 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.854932 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.855336 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.855411 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.855420 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.855609 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.855658 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:22:54.856055 6434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112f14f0b49d484b8570e101d33fb66b" candidate_uuid: "1cf3c56e80e94739a449813296fb5f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c55e028888437e9969d643eff2f5de" is_pre_election: true
I20260808 02:22:54.856060 6153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112f14f0b49d484b8570e101d33fb66b" candidate_uuid: "1cf3c56e80e94739a449813296fb5f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" is_pre_election: true
I20260808 02:22:54.856204 6153 raft_consensus.cc:2454] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf3c56e80e94739a449813296fb5f2e in term 0.
I20260808 02:22:54.856204 6434 raft_consensus.cc:2454] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf3c56e80e94739a449813296fb5f2e in term 0.
I20260808 02:22:54.856437 6227 leader_election.cc:302] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf3c56e80e94739a449813296fb5f2e, b0c55e028888437e9969d643eff2f5de; no voters:
I20260808 02:22:54.856598 6520 raft_consensus.cc:2781] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:54.856676 6520 raft_consensus.cc:482] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:54.856694 6520 raft_consensus.cc:3037] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:54.857410 6520 raft_consensus.cc:504] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.857548 6520 leader_election.cc:288] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [CANDIDATE]: Term 1 election: Requested vote from peers 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219), b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837)
I20260808 02:22:54.857709 6153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112f14f0b49d484b8570e101d33fb66b" candidate_uuid: "1cf3c56e80e94739a449813296fb5f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3"
I20260808 02:22:54.857731 6434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112f14f0b49d484b8570e101d33fb66b" candidate_uuid: "1cf3c56e80e94739a449813296fb5f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c55e028888437e9969d643eff2f5de"
I20260808 02:22:54.857803 6153 raft_consensus.cc:3037] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:54.857854 6434 raft_consensus.cc:3037] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:54.858652 6434 raft_consensus.cc:2454] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf3c56e80e94739a449813296fb5f2e in term 1.
I20260808 02:22:54.858721 6153 raft_consensus.cc:2454] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf3c56e80e94739a449813296fb5f2e in term 1.
I20260808 02:22:54.858886 6227 leader_election.cc:302] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf3c56e80e94739a449813296fb5f2e, 3177ad2f6fcb4ef0962396eb5ad844d3; no voters:
I20260808 02:22:54.858994 6520 raft_consensus.cc:2781] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:54.859154 6520 raft_consensus.cc:686] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 1 LEADER]: Becoming Leader. State: Replica: 1cf3c56e80e94739a449813296fb5f2e, State: Running, Role: LEADER
I20260808 02:22:54.859262 6520 consensus_queue.cc:237] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } }
I20260808 02:22:54.860096 5165 catalog_manager.cc:5495] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2). New cstate: current_term: 1 leader_uuid: "1cf3c56e80e94739a449813296fb5f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:54.889192 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.886949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.30.160.2:34901)
0808 02:22:54.887011 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.887013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.887022 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:22:54.887063 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.887064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.887083 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.887145 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:54.887262 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.887749 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.888508 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.888687 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.888737 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.888879 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:54.888979 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:54.889041 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.889063 (+ 22us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.889080 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:22:54.891047 6434 raft_consensus.cc:1261] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Refusing update from remote peer 1cf3c56e80e94739a449813296fb5f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:54.891047 6153 raft_consensus.cc:1261] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 FOLLOWER]: Refusing update from remote peer 1cf3c56e80e94739a449813296fb5f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:54.891430 6520 consensus_queue.cc:1035] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:54.891566 6522 consensus_queue.cc:1035] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:54.894562 6526 mvcc.cc:205] Tried to move back new op lower bound from 7316094053950881792 to 7316094053824032768. Current Snapshot: MvccSnapshot[applied={T|T < 7316094053950881792}]
I20260808 02:22:54.895706 6527 mvcc.cc:205] Tried to move back new op lower bound from 7316094053950881792 to 7316094053824032768. Current Snapshot: MvccSnapshot[applied={T|T < 7316094053950881792}]
I20260808 02:22:54.897231 6525 mvcc.cc:205] Tried to move back new op lower bound from 7316094053950881792 to 7316094053824032768. Current Snapshot: MvccSnapshot[applied={T|T < 7316094053950881792 or (T in {7316094053950881792})}]
I20260808 02:22:54.897545 5165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:54.897636 5165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:54.898973 5165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbceee63e42341518b449bd002318f22: Sending DeleteTablet for 3 replicas of tablet 112f14f0b49d484b8570e101d33fb66b
I20260808 02:22:54.899287 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 112f14f0b49d484b8570e101d33fb66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:54 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:47606
I20260808 02:22:54.899452 6414 tablet_service.cc:1514] Processing DeleteTablet for tablet 112f14f0b49d484b8570e101d33fb66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:54 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:37092
I20260808 02:22:54.899504 6131 tablet_service.cc:1514] Processing DeleteTablet for tablet 112f14f0b49d484b8570e101d33fb66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:54 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:48288
I20260808 02:22:54.899766 6535 tablet_replica.cc:330] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: stopping tablet replica
I20260808 02:22:54.899741 6534 tablet_replica.cc:330] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: stopping tablet replica
I20260808 02:22:54.899925 6535 raft_consensus.cc:2229] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:54.899930 6534 raft_consensus.cc:2229] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:22:54.900039 6534 raft_consensus.cc:2258] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:54.900061 6535 raft_consensus.cc:2258] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:54.900490 6534 ts_tablet_manager.cc:1905] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:54.900779 6535 ts_tablet_manager.cc:1905] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:54.901005 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5131
I20260808 02:22:54.903163 6534 ts_tablet_manager.cc:1918] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:54.903244 6534 log.cc:1196] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/112f14f0b49d484b8570e101d33fb66b
I20260808 02:22:54.903565 6534 ts_tablet_manager.cc:1939] T 112f14f0b49d484b8570e101d33fb66b P 1cf3c56e80e94739a449813296fb5f2e: Deleting consensus metadata
I20260808 02:22:54.904476 6535 ts_tablet_manager.cc:1918] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:54.904618 6535 log.cc:1196] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/112f14f0b49d484b8570e101d33fb66b
I20260808 02:22:54.904960 6535 ts_tablet_manager.cc:1939] T 112f14f0b49d484b8570e101d33fb66b P b0c55e028888437e9969d643eff2f5de: Deleting consensus metadata
I20260808 02:22:54.908259 6536 tablet_replica.cc:330] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: stopping tablet replica
I20260808 02:22:54.908424 6536 raft_consensus.cc:2229] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:54.908576 6536 raft_consensus.cc:2258] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:22:54.909034 6368 connection.cc:535] server connection from 127.0.0.1:37092 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:22:54.909144 6536 ts_tablet_manager.cc:1905] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:22:54.909205 6225 connection.cc:535] server connection from 127.0.0.1:47606 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:22:54.910828 6536 ts_tablet_manager.cc:1918] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:54.910960 6536 log.cc:1196] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/112f14f0b49d484b8570e101d33fb66b
I20260808 02:22:54.911301 6536 ts_tablet_manager.cc:1939] T 112f14f0b49d484b8570e101d33fb66b P 3177ad2f6fcb4ef0962396eb5ad844d3: Deleting consensus metadata
W20260808 02:22:54.911731 6078 connection.cc:441] server connection from 127.0.0.1:48288 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48288 (request call id 1) could send its response
I20260808 02:22:54.916306 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.900125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.30.160.60:35609)
0808 02:22:54.900171 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.900174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.908087 (+ 7913us) server_negotiation.cc:407] Connection header received
0808 02:22:54.908203 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.908204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.908218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.908279 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.909531 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.910293 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.911333 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.911566 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.911644 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.913230 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.913232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.913232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.913240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.915028 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.915437 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.915438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.915438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.915508 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.915715 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.915716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.915716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.915853 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.915970 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.916120 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.916208 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:54.920011 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.912275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.30.160.61:38335)
0808 02:22:54.912335 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.912337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.912738 (+ 401us) server_negotiation.cc:407] Connection header received
0808 02:22:54.912814 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.912815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.912829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.912885 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.913420 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.914131 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.915533 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.915834 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.915879 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.917464 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.917465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.917466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.917472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.918760 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.919126 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.919127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.919128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.919188 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.919400 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.919401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.919402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.919531 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.919626 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.919751 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.919866 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:54.926173 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:54.912519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.30.160.60:35609)
0808 02:22:54.912557 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:54.912559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:54.919996 (+ 7437us) server_negotiation.cc:407] Connection header received
0808 02:22:54.920585 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:54.920586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:54.920595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:54.920645 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:54.921137 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.921648 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.922371 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:54.922573 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:54.922646 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:54.923708 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:54.923709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:54.923710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:54.923717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:54.925096 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.925427 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.925428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.925429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.925481 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:54.925689 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:54.925690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:54.925691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:54.925798 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:54.925903 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:54.926008 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 02:22:54.926077 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:55.896802 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:55.896465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:44447)
0808 02:22:55.896642 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:55.896737 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
W20260808 02:22:55.897063 6484 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:36979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:55.897861 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:55.897648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:43227)
0808 02:22:55.897753 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:55.897826 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
W20260808 02:22:55.898156 6345 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:36979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:55.899796 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:55.899615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:34257)
0808 02:22:55.899701 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:55.899760 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":45,"threads_started":1}
W20260808 02:22:55.900018 6202 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:36979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.400043 6548 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbceee63e42341518b449bd002318f22)
I20260808 02:22:56.400125 6549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbceee63e42341518b449bd002318f22)
I20260808 02:22:56.400220 6548 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:56.400231 6549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:56.400544 6549 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbceee63e42341518b449bd002318f22 (127.30.160.62:36979), f0f0d4a6c0144573955a055e9eb58caa (127.30.160.61:38335)
I20260808 02:22:56.400543 6548 leader_election.cc:288] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbceee63e42341518b449bd002318f22 (127.30.160.62:36979), 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60:35609)
I20260808 02:22:56.401002 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.400691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33538)
0808 02:22:56.400885 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.400951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":55,"threads_started":1}
W20260808 02:22:56.401211 5794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.401276 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.400880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33546)
0808 02:22:56.401171 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.401225 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":57,"threads_started":1}
W20260808 02:22:56.401464 5995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.401686 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.401570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33554)
0808 02:22:56.401629 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.401660 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 02:22:56.401854 5794 leader_election.cc:334] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.401939 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.401857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33570)
0808 02:22:56.401890 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.401913 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260808 02:22:56.402014 5995 leader_election.cc:334] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.403983 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.400780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:35609 (local address 127.0.0.1:56712)
0808 02:22:56.400947 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.400959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:56.401027 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:56.401307 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:56.401308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:56.401311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:56.401383 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:56.401385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.402406 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.402407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:56.403171 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:56.403174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.403704 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.403705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:56.403764 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:56.403767 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:56.403825 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:22:56.403887 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260808 02:22:56.403986 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.401061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.30.160.60:35609)
0808 02:22:56.401154 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:56.401157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:56.401164 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:56.401192 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:56.401193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:56.401202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:56.401450 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:56.401646 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.402321 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.403268 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.403626 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.403680 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:56.403689 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:56.403845 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:22:56.403913 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
I20260808 02:22:56.403996 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.400711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:38335 (local address 127.0.0.1:33674)
0808 02:22:56.400987 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.400998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:56.401064 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:56.401434 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:56.401435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:56.401438 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:56.401531 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:56.401534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.402444 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.402445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:56.403168 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:56.403170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.403748 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.403749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:56.403791 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:56.403794 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:56.403873 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:22:56.403929 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260808 02:22:56.404043 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.400883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.30.160.61:38335)
0808 02:22:56.401181 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:56.401183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:56.401191 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:56.401215 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:56.401216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:56.401226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:56.401623 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:56.401663 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.402328 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.403272 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:56.403642 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:56.403725 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:56.403735 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:56.403908 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:22:56.403946 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":32,"threads_started":1}
I20260808 02:22:56.404430 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a2e5751a1684453bbc957d4f3e85290" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0f0d4a6c0144573955a055e9eb58caa" is_pre_election: true
I20260808 02:22:56.404546 5819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2e5751a1684453bbc957d4f3e85290 in term 1.
I20260808 02:22:56.404536 6020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0f0d4a6c0144573955a055e9eb58caa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a2e5751a1684453bbc957d4f3e85290" is_pre_election: true
I20260808 02:22:56.404673 6020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f0d4a6c0144573955a055e9eb58caa in term 1.
I20260808 02:22:56.404714 5993 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a2e5751a1684453bbc957d4f3e85290, f0f0d4a6c0144573955a055e9eb58caa; no voters: bbceee63e42341518b449bd002318f22
I20260808 02:22:56.404819 5793 leader_election.cc:302] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a2e5751a1684453bbc957d4f3e85290, f0f0d4a6c0144573955a055e9eb58caa; no voters: bbceee63e42341518b449bd002318f22
I20260808 02:22:56.404819 6549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 02:22:56.404874 6549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbceee63e42341518b449bd002318f22)
I20260808 02:22:56.404910 6549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:22:56.404932 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 02:22:56.404973 6548 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbceee63e42341518b449bd002318f22)
I20260808 02:22:56.404997 6548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:22:56.405625 6549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:56.405648 6548 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:56.405768 6549 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [CANDIDATE]: Term 2 election: Requested vote from peers bbceee63e42341518b449bd002318f22 (127.30.160.62:36979), f0f0d4a6c0144573955a055e9eb58caa (127.30.160.61:38335)
I20260808 02:22:56.405781 6548 leader_election.cc:288] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 2 election: Requested vote from peers bbceee63e42341518b449bd002318f22 (127.30.160.62:36979), 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60:35609)
I20260808 02:22:56.405980 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.405865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33578)
0808 02:22:56.405899 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.405935 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260808 02:22:56.406080 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.405928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33590)
0808 02:22:56.405989 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.406017 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 02:22:56.406093 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a2e5751a1684453bbc957d4f3e85290" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0f0d4a6c0144573955a055e9eb58caa"
I20260808 02:22:56.406116 6020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0f0d4a6c0144573955a055e9eb58caa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a2e5751a1684453bbc957d4f3e85290"
I20260808 02:22:56.406173 5819 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a2e5751a1684453bbc957d4f3e85290 in current term 2: Already voted for candidate f0f0d4a6c0144573955a055e9eb58caa in this term.
I20260808 02:22:56.406198 6020 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0f0d4a6c0144573955a055e9eb58caa in current term 2: Already voted for candidate 3a2e5751a1684453bbc957d4f3e85290 in this term.
I20260808 02:22:56.406878 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.406775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33608)
0808 02:22:56.406829 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.406852 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 02:22:56.406939 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.406854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33624)
0808 02:22:56.406887 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.406913 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260808 02:22:56.406942 5995 leader_election.cc:334] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.406993 5995 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a2e5751a1684453bbc957d4f3e85290; no voters: bbceee63e42341518b449bd002318f22, f0f0d4a6c0144573955a055e9eb58caa
W20260808 02:22:56.407003 5794 leader_election.cc:334] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:56.407050 5794 leader_election.cc:302] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0f0d4a6c0144573955a055e9eb58caa; no voters: 3a2e5751a1684453bbc957d4f3e85290, bbceee63e42341518b449bd002318f22
I20260808 02:22:56.407124 6548 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 02:22:56.407220 6549 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 02:22:56.897599 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.897362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:42033)
0808 02:22:56.897506 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.897552 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260808 02:22:56.898765 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.898531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:46363)
0808 02:22:56.898645 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.898679 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260808 02:22:56.900445 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:56.900288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:60109)
0808 02:22:56.900389 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:56.900411 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260808 02:22:57.898262 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.898039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:34877)
0808 02:22:57.898159 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.898214 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260808 02:22:57.899399 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.899199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:50975)
0808 02:22:57.899311 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.899354 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260808 02:22:57.900995 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.900823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:42913)
0808 02:22:57.900926 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.900958 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260808 02:22:57.952373 6564 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:57.952463 6564 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:57.952631 6564 leader_election.cc:288] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbceee63e42341518b449bd002318f22 (127.30.160.62:36979), 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60:35609)
I20260808 02:22:57.952890 6020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0f0d4a6c0144573955a055e9eb58caa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a2e5751a1684453bbc957d4f3e85290" is_pre_election: true
I20260808 02:22:57.952970 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.952732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33802)
0808 02:22:57.952888 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.952929 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260808 02:22:57.952997 6020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f0d4a6c0144573955a055e9eb58caa in term 2.
I20260808 02:22:57.953166 5793 leader_election.cc:302] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2e5751a1684453bbc957d4f3e85290, f0f0d4a6c0144573955a055e9eb58caa; no voters:
I20260808 02:22:57.953290 6564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 02:22:57.953349 6564 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:57.953374 6564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:22:57.953450 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.953329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33808)
0808 02:22:57.953389 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.953425 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 02:22:57.953557 5794 leader_election.cc:334] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:57.954195 6564 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:57.954341 6564 leader_election.cc:288] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 3 election: Requested vote from peers bbceee63e42341518b449bd002318f22 (127.30.160.62:36979), 3a2e5751a1684453bbc957d4f3e85290 (127.30.160.60:35609)
I20260808 02:22:57.954483 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.954402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33816)
0808 02:22:57.954435 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.954459 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 02:22:57.954553 6020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0f0d4a6c0144573955a055e9eb58caa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a2e5751a1684453bbc957d4f3e85290"
I20260808 02:22:57.954646 6020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:22:57.954823 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:57.954747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33832)
0808 02:22:57.954778 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:57.954801 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260808 02:22:57.954905 5794 leader_election.cc:334] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
I20260808 02:22:57.955421 6020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f0d4a6c0144573955a055e9eb58caa in term 3.
I20260808 02:22:57.955591 5793 leader_election.cc:302] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a2e5751a1684453bbc957d4f3e85290, f0f0d4a6c0144573955a055e9eb58caa; no voters: bbceee63e42341518b449bd002318f22
I20260808 02:22:57.955698 6564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:22:57.955802 6564 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [term 3 LEADER]: Becoming Leader. State: Replica: f0f0d4a6c0144573955a055e9eb58caa, State: Running, Role: LEADER
I20260808 02:22:57.955886 6564 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } }
I20260808 02:22:57.956283 6567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f0d4a6c0144573955a055e9eb58caa. Latest consensus state: current_term: 3 leader_uuid: "f0f0d4a6c0144573955a055e9eb58caa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:57.956394 6567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:57.956588 6569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:57.957042 6569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:57.957105 6569 catalog_manager.cc:1259] Loaded cluster ID: 15468bf169d34060b08eb4005209c309
I20260808 02:22:57.957158 6569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:57.957352 6569 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:57.957448 6569 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Loaded TSK: 0
I20260808 02:22:57.957602 6569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:58.012749 5808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:22:58.014392 5808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:22:58.015542 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.015435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33864)
0808 02:22:58.015482 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.015515 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 02:22:58.015565 6020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [term 3 FOLLOWER]: Refusing update from remote peer f0f0d4a6c0144573955a055e9eb58caa: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 02:22:58.015830 6564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 02:22:58.015998 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.015880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.0.0.1:33868)
0808 02:22:58.015938 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.015964 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 02:22:58.016122 5794 consensus_peers.cc:487] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa -> Peer bbceee63e42341518b449bd002318f22 (127.30.160.62:36979): Couldn't send request to peer bbceee63e42341518b449bd002318f22. Status: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:22:58.016573 6572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f0d4a6c0144573955a055e9eb58caa. Latest consensus state: current_term: 3 leader_uuid: "f0f0d4a6c0144573955a055e9eb58caa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:58.016688 6572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:58.017103 6570 mvcc.cc:205] Tried to move back new op lower bound from 7316094066749321216 to 7316094066508218368. Current Snapshot: MvccSnapshot[applied={T|T < 7316094066749321216}]
I20260808 02:22:58.017220 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f0f0d4a6c0144573955a055e9eb58caa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:58.017342 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:58.017361 6573 mvcc.cc:205] Tried to move back new op lower bound from 7316094066749321216 to 7316094066508218368. Current Snapshot: MvccSnapshot[applied={T|T < 7316094066749321216}]
I20260808 02:22:58.017544 6572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f0f0d4a6c0144573955a055e9eb58caa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bbceee63e42341518b449bd002318f22" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 36979 } } peers { permanent_uuid: "f0f0d4a6c0144573955a055e9eb58caa" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 38335 } attrs { promote: false } } peers { permanent_uuid: "3a2e5751a1684453bbc957d4f3e85290" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 35609 } attrs { promote: false } } }
I20260808 02:22:58.017630 6572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e5751a1684453bbc957d4f3e85290 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:58.022804 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.019259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:34901 (local address 127.0.0.1:47610)
0808 02:22:58.019510 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.019521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:58.019564 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:58.019811 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:58.019811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:58.019824 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:58.019865 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.019866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.020825 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.020825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.021796 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.021799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.022542 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.022543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.022606 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.022611 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:58.022673 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:22:58.022723 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260808 02:22:58.022827 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.019399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.30.160.2:34901)
0808 02:22:58.019655 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:58.019657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:58.019665 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:58.019691 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:58.019692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:58.019702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:58.019764 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:58.019972 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.020751 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.021914 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.022456 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.022529 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.022542 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:58.022691 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:22:58.022746 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":64,"threads_started":1}
I20260808 02:22:58.023217 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.019452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.30.160.1:39219)
0808 02:22:58.019546 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:58.019548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:58.019556 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:58.019588 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:58.019589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:58.019598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:58.019689 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:58.019824 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.020675 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.022099 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.022676 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.023062 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.023081 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:58.023102 (+ 21us) server_negotiation.cc:299] Negotiation successful
0808 02:22:58.023151 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
I20260808 02:22:58.023283 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.019384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.30.160.3:34837)
0808 02:22:58.019551 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:58.019554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:58.019562 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:58.019589 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:58.019590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:58.019602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:58.019829 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:58.019871 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.020709 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.022104 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.022682 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.023130 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.023145 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:58.023164 (+ 19us) server_negotiation.cc:299] Negotiation successful
0808 02:22:58.023208 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260808 02:22:58.023011 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.019248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34837 (local address 127.0.0.1:37106)
0808 02:22:58.019399 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.019410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:58.019455 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:58.019701 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:58.019702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:58.019704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:58.019763 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.019765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.020818 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.020819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.021995 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.021998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.022769 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.022770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.022827 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.022832 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:58.022863 (+ 31us) client_negotiation.cc:240] Negotiation successful
0808 02:22:58.022911 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260808 02:22:58.023077 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.019148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39219 (local address 127.0.0.1:48294)
0808 02:22:58.019177 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.019188 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:58.019237 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:58.019692 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:58.019693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:58.019695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:58.019749 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.019750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.020761 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.020762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.022018 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.022020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.022775 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.022776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.022841 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.022846 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:22:58.022879 (+ 33us) client_negotiation.cc:240] Negotiation successful
0808 02:22:58.022928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":51}
I20260808 02:22:58.023978 6276 tablet_service.cc:1467] Processing CreateTablet for tablet e729dfb5f7434adc86fcb5b5e74c782c (DEFAULT_TABLE table=test-table [id=76897c6a31cd4c7493ca8db6cb1a1d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:58.023996 6131 tablet_service.cc:1467] Processing CreateTablet for tablet e729dfb5f7434adc86fcb5b5e74c782c (DEFAULT_TABLE table=test-table [id=76897c6a31cd4c7493ca8db6cb1a1d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:58.024011 6414 tablet_service.cc:1467] Processing CreateTablet for tablet e729dfb5f7434adc86fcb5b5e74c782c (DEFAULT_TABLE table=test-table [id=76897c6a31cd4c7493ca8db6cb1a1d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:22:58.024125 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e729dfb5f7434adc86fcb5b5e74c782c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:58.024125 6414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e729dfb5f7434adc86fcb5b5e74c782c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:58.024125 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e729dfb5f7434adc86fcb5b5e74c782c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:58.025206 6582 tablet_bootstrap.cc:492] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3: Bootstrap starting.
I20260808 02:22:58.025581 6582 tablet_bootstrap.cc:654] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:58.025652 6584 tablet_bootstrap.cc:492] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: Bootstrap starting.
I20260808 02:22:58.026118 6584 tablet_bootstrap.cc:654] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:58.026360 6582 tablet_bootstrap.cc:492] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3: No bootstrap required, opened a new log
I20260808 02:22:58.026423 6582 ts_tablet_manager.cc:1397] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:58.026432 6583 tablet_bootstrap.cc:492] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Bootstrap starting.
I20260808 02:22:58.026628 6582 raft_consensus.cc:348] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.026713 6584 tablet_bootstrap.cc:492] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: No bootstrap required, opened a new log
I20260808 02:22:58.026742 6582 raft_consensus.cc:374] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:58.026782 6584 ts_tablet_manager.cc:1397] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:58.026791 6582 raft_consensus.cc:729] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3177ad2f6fcb4ef0962396eb5ad844d3, State: Initialized, Role: FOLLOWER
I20260808 02:22:58.026808 6583 tablet_bootstrap.cc:654] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:58.026924 6582 consensus_queue.cc:260] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.026939 6584 raft_consensus.cc:348] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.027007 6584 raft_consensus.cc:374] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:58.027052 6584 raft_consensus.cc:729] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf3c56e80e94739a449813296fb5f2e, State: Initialized, Role: FOLLOWER
I20260808 02:22:58.027163 6582 ts_tablet_manager.cc:1428] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:58.027144 6584 consensus_queue.cc:260] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.027323 6584 ts_tablet_manager.cc:1428] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:58.027431 6583 tablet_bootstrap.cc:492] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: No bootstrap required, opened a new log
I20260808 02:22:58.027487 6583 ts_tablet_manager.cc:1397] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:58.027606 6583 raft_consensus.cc:348] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.027669 6583 raft_consensus.cc:374] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:58.027688 6583 raft_consensus.cc:729] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c55e028888437e9969d643eff2f5de, State: Initialized, Role: FOLLOWER
I20260808 02:22:58.027693 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.027570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:51253)
0808 02:22:58.027640 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.027667 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 02:22:58.027731 6583 consensus_queue.cc:260] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.027941 6583 ts_tablet_manager.cc:1428] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:58.028177 6346 heartbeater.cc:503] Master 127.30.160.61:38335 was elected leader, sending a full tablet report...
I20260808 02:22:58.028256 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.027396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:52199)
0808 02:22:58.028194 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.028225 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779}
I20260808 02:22:58.028290 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.028186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:43293)
0808 02:22:58.028232 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.028257 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 02:22:58.029347 6203 heartbeater.cc:503] Master 127.30.160.61:38335 was elected leader, sending a full tablet report...
I20260808 02:22:58.029407 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.029292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:53051)
0808 02:22:58.029329 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.029360 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260808 02:22:58.029652 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.029564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:37171)
0808 02:22:58.029604 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.029627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 02:22:58.029801 6485 heartbeater.cc:503] Master 127.30.160.61:38335 was elected leader, sending a full tablet report...
I20260808 02:22:58.180061 6588 raft_consensus.cc:482] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:22:58.180168 6588 raft_consensus.cc:504] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.180501 6588 leader_election.cc:288] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837), 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2:34901)
I20260808 02:22:58.183511 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.180766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:44129 (local address 127.30.160.2:34901)
0808 02:22:58.180834 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:58.180836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:58.181011 (+ 175us) server_negotiation.cc:407] Connection header received
0808 02:22:58.181041 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:58.181042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:58.181052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:58.181107 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:58.181280 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.181757 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.182735 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.183123 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.183212 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.183224 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:58.183389 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:22:58.183434 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:22:58.183521 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.180790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:60355 (local address 127.30.160.3:34837)
0808 02:22:58.180873 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:58.180875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:58.180886 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:58.180920 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:58.180921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:58.180933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:58.180992 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:22:58.181191 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.181704 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.182759 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.183127 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.183214 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.183227 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:58.183409 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:22:58.183453 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:22:58.183532 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.180651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34837 (local address 127.30.160.1:60355)
0808 02:22:58.180735 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.180750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:58.180814 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:58.181009 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:58.181011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:58.181014 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:58.181109 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.181111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.181789 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.181790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.182677 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.182681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.183233 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.183234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.183317 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.183321 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:22:58.183397 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:22:58.183439 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":40}
I20260808 02:22:58.183553 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.180691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:34901 (local address 127.30.160.1:44129)
0808 02:22:58.180896 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.180906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:58.180972 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:58.181119 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:58.181120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:58.181121 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:22:58.181212 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.181214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.181847 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.181848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.182633 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:58.182636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.183211 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.183213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:58.183289 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.183292 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:22:58.183377 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:22:58.183421 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":121,"threads_started":1}
I20260808 02:22:58.183820 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e729dfb5f7434adc86fcb5b5e74c782c" candidate_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf3c56e80e94739a449813296fb5f2e" is_pre_election: true
I20260808 02:22:58.184000 6297 raft_consensus.cc:2454] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3177ad2f6fcb4ef0962396eb5ad844d3 in term 0.
I20260808 02:22:58.184015 6434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e729dfb5f7434adc86fcb5b5e74c782c" candidate_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c55e028888437e9969d643eff2f5de" is_pre_election: true
I20260808 02:22:58.184118 6434 raft_consensus.cc:2454] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3177ad2f6fcb4ef0962396eb5ad844d3 in term 0.
I20260808 02:22:58.184207 6079 leader_election.cc:302] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf3c56e80e94739a449813296fb5f2e, 3177ad2f6fcb4ef0962396eb5ad844d3; no voters:
I20260808 02:22:58.184353 6588 raft_consensus.cc:2781] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:22:58.184418 6588 raft_consensus.cc:482] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:22:58.184449 6588 raft_consensus.cc:3037] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:58.185161 6588 raft_consensus.cc:504] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.185320 6588 leader_election.cc:288] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [CANDIDATE]: Term 1 election: Requested vote from peers b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837), 1cf3c56e80e94739a449813296fb5f2e (127.30.160.2:34901)
I20260808 02:22:58.185431 6434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e729dfb5f7434adc86fcb5b5e74c782c" candidate_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c55e028888437e9969d643eff2f5de"
I20260808 02:22:58.185536 6434 raft_consensus.cc:3037] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:58.185509 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e729dfb5f7434adc86fcb5b5e74c782c" candidate_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf3c56e80e94739a449813296fb5f2e"
I20260808 02:22:58.185583 6297 raft_consensus.cc:3037] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:58.186295 6297 raft_consensus.cc:2454] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3177ad2f6fcb4ef0962396eb5ad844d3 in term 1.
I20260808 02:22:58.186329 6434 raft_consensus.cc:2454] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3177ad2f6fcb4ef0962396eb5ad844d3 in term 1.
I20260808 02:22:58.186515 6080 leader_election.cc:302] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3177ad2f6fcb4ef0962396eb5ad844d3, b0c55e028888437e9969d643eff2f5de; no voters:
I20260808 02:22:58.186619 6588 raft_consensus.cc:2781] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:58.186810 6588 raft_consensus.cc:686] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3177ad2f6fcb4ef0962396eb5ad844d3, State: Running, Role: LEADER
I20260808 02:22:58.186866 6588 consensus_queue.cc:237] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } }
I20260808 02:22:58.187223 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.187068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:52113)
0808 02:22:58.187157 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.187193 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 02:22:58.187784 5807 catalog_manager.cc:5495] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1). New cstate: current_term: 1 leader_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3177ad2f6fcb4ef0962396eb5ad844d3" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 } health_report { overall_health: UNKNOWN } } }
I20260808 02:22:58.204514 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.202388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.30.160.1:39219)
0808 02:22:58.202440 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:58.202442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:58.202450 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:58.202493 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:58.202493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:58.202510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:58.202574 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:58.202682 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.203140 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.203817 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:58.204068 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:58.204124 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:58.204253 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:58.204348 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:58.204404 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:58.204444 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 02:22:58.204462 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:22:58.205534 6434 raft_consensus.cc:1261] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Refusing update from remote peer 3177ad2f6fcb4ef0962396eb5ad844d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:58.205647 6297 raft_consensus.cc:1261] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Refusing update from remote peer 3177ad2f6fcb4ef0962396eb5ad844d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:58.205817 6592 consensus_queue.cc:1035] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c55e028888437e9969d643eff2f5de" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:58.205943 6588 consensus_queue.cc:1035] T e729dfb5f7434adc86fcb5b5e74c782c P 3177ad2f6fcb4ef0962396eb5ad844d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf3c56e80e94739a449813296fb5f2e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 34901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:58.207264 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.206944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:49359)
0808 02:22:58.207150 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.207184 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260808 02:22:58.207309 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.207153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:42705)
0808 02:22:58.207226 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.207272 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260808 02:22:58.208526 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.208433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:51765)
0808 02:22:58.208472 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.208498 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 02:22:58.209067 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.208978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.3:46359)
0808 02:22:58.209016 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.209042 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 02:22:58.209357 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.208734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.1:43357)
0808 02:22:58.209302 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.209333 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552}
I20260808 02:22:58.209671 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.209123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:36979 (local address 127.30.160.2:38773)
0808 02:22:58.209575 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.209602 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:36979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
I20260808 02:22:58.211229 5807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:58.211337 5807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:22:58.212626 5807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0f0d4a6c0144573955a055e9eb58caa: Sending DeleteTablet for 3 replicas of tablet e729dfb5f7434adc86fcb5b5e74c782c
I20260808 02:22:58.212932 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet e729dfb5f7434adc86fcb5b5e74c782c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:58 UTC) from {username='oryx', principal='oryx/127.30.160.61@KRBTEST.COM'} at 127.0.0.1:47610
I20260808 02:22:58.213109 6607 tablet_replica.cc:330] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: stopping tablet replica
I20260808 02:22:58.213186 6607 raft_consensus.cc:2229] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:58.213243 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5964
I20260808 02:22:58.213266 6607 raft_consensus.cc:2258] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:58.213367 6414 tablet_service.cc:1514] Processing DeleteTablet for tablet e729dfb5f7434adc86fcb5b5e74c782c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:22:58 UTC) from {username='oryx', principal='oryx/127.30.160.61@KRBTEST.COM'} at 127.0.0.1:37106
I20260808 02:22:58.215787 6608 tablet_replica.cc:330] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: stopping tablet replica
I20260808 02:22:58.215869 6608 raft_consensus.cc:2229] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:22:58.215966 6608 raft_consensus.cc:2258] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:22:58.216287 6608 ts_tablet_manager.cc:1905] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:22:58.217753 6608 ts_tablet_manager.cc:1918] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:22:58.217855 6608 log.cc:1196] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/e729dfb5f7434adc86fcb5b5e74c782c
I20260808 02:22:58.218071 6608 ts_tablet_manager.cc:1939] T e729dfb5f7434adc86fcb5b5e74c782c P b0c55e028888437e9969d643eff2f5de: Deleting consensus metadata
I20260808 02:22:58.218393 5793 catalog_manager.cc:4841] TS b0c55e028888437e9969d643eff2f5de (127.30.160.3:34837): tablet e729dfb5f7434adc86fcb5b5e74c782c (table test-table [id=76897c6a31cd4c7493ca8db6cb1a1d2d]) successfully deleted
I20260808 02:22:58.223699 6607 ts_tablet_manager.cc:1905] T e729dfb5f7434adc86fcb5b5e74c782c P 1cf3c56e80e94739a449813296fb5f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:22:58.224455 5793 connection.cc:535] client connection to 127.30.160.1:39219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:22:58.224938 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.224822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39219 (local address 127.0.0.1:48326)
0808 02:22:58.224874 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.224899 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:22:58.224998 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 6209
W20260808 02:22:58.225111 5793 catalog_manager.cc:4584] TS 3177ad2f6fcb4ef0962396eb5ad844d3 (127.30.160.1:39219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e729dfb5f7434adc86fcb5b5e74c782c: Network error: Client connection negotiation failed: client connection to 127.30.160.1:39219: connect: Connection refused (error 111)
I20260808 02:22:58.232962 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 6351
I20260808 02:22:58.233129 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:58.232974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:34901 (local address 127.0.0.1:47618)
0808 02:22:58.233057 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:58.233091 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.2:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:22:58.239090 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5666
I20260808 02:22:58.245487 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 5866
2026-08-08T02:22:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6487 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:58 dist-test-slave-zkpd krb5kdc[6611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:58 dist-test-slave-zkpd krb5kdc[6611](info): set up 2 sockets
Aug 08 02:22:58 dist-test-slave-zkpd krb5kdc[6611](info): commencing operation
krb5kdc: starting...
W20260808 02:22:59.305119 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab.
Entry for principal oryx/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab.
I20260808 02:22:59.427080 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:46401
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab
--principal=oryx/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:46401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:59.513279 6624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:59.513522 6624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:59.513556 6624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:59.514734 6624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:59.514806 6624 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:22:59.514839 6624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:59.514873 6624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:59.514896 6624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:59.514923 6624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:59.516484 6624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:46401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.62.keytab
--principal=oryx/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:46401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:59.516769 6624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:59.517046 6624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:59.521842 6632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:59.521811 6630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:59.521816 6629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:59.522156 6624 server_base.cc:1034] running on GCE node
I20260808 02:22:59.522419 6624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:59.522714 6624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:59.523926 6624 hybrid_clock.cc:648] HybridClock initialized: now 1786155779523886 us; error 55 us; skew 500 ppm
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:59.526319 6624 init.cc:377] Logged in from keytab as oryx/127.30.160.62@KRBTEST.COM (short username oryx)
I20260808 02:22:59.526743 6624 webserver.cc:460] Webserver started at http://127.30.160.62:37497/ using document root <none> and password file <none>
I20260808 02:22:59.526988 6624 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:59.527056 6624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:59.527204 6624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:59.528368 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "2daa45bf98af432a8f1e771d88008f3f"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "985ec7f40dfa2af6b1e84eb88bd6cab5"
server_key_iv: "416eba71bf557ceed1d40ef7410e60f8"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.528738 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "2daa45bf98af432a8f1e771d88008f3f"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "985ec7f40dfa2af6b1e84eb88bd6cab5"
server_key_iv: "416eba71bf557ceed1d40ef7410e60f8"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.530056 6624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:59.530740 6639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:59.530933 6624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:59.531041 6624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "2daa45bf98af432a8f1e771d88008f3f"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "985ec7f40dfa2af6b1e84eb88bd6cab5"
server_key_iv: "416eba71bf557ceed1d40ef7410e60f8"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.531117 6624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:59.544485 6624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:59.546649 6624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:59.546833 6624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:59.551592 6624 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:46401
I20260808 02:22:59.551636 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:46401 every 8 connection(s)
I20260808 02:22:59.552083 6624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:59.552836 6705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:59.555135 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Bootstrap starting.
I20260808 02:22:59.555697 6705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:59.556044 6705 log.cc:824] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:59.556715 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: No bootstrap required, opened a new log
I20260808 02:22:59.558720 6705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daa45bf98af432a8f1e771d88008f3f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 46401 } }
I20260808 02:22:59.558871 6705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:59.558894 6705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2daa45bf98af432a8f1e771d88008f3f, State: Initialized, Role: FOLLOWER
I20260808 02:22:59.559031 6705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daa45bf98af432a8f1e771d88008f3f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 46401 } }
I20260808 02:22:59.559113 6705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:59.559137 6705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:59.559165 6705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:59.559646 6705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daa45bf98af432a8f1e771d88008f3f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 46401 } }
I20260808 02:22:59.559734 6705 leader_election.cc:302] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2daa45bf98af432a8f1e771d88008f3f; no voters:
I20260808 02:22:59.559861 6705 leader_election.cc:288] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:59.559996 6708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:59.560173 6708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [term 1 LEADER]: Becoming Leader. State: Replica: 2daa45bf98af432a8f1e771d88008f3f, State: Running, Role: LEADER
I20260808 02:22:59.560176 6705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:59.560319 6708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daa45bf98af432a8f1e771d88008f3f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 46401 } }
I20260808 02:22:59.560786 6709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2daa45bf98af432a8f1e771d88008f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daa45bf98af432a8f1e771d88008f3f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 46401 } } }
I20260808 02:22:59.560877 6709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:59.561020 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2daa45bf98af432a8f1e771d88008f3f. Latest consensus state: current_term: 1 leader_uuid: "2daa45bf98af432a8f1e771d88008f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daa45bf98af432a8f1e771d88008f3f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 46401 } } }
I20260808 02:22:59.561107 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:59.561403 6717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:59.561872 6717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:59.562024 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 6624
I20260808 02:22:59.562120 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:22:59.562356 31360 external_mini_cluster.cc:1428] Setting key b274edde27d000dc9bc26492a1fce09f
I20260808 02:22:59.563820 6717 catalog_manager.cc:1347] Generated new cluster ID: 0c1bcdc57ccb427d9684a569403134d8
I20260808 02:22:59.563875 6717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:59.574257 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.562853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.30.160.62:46401)
0808 02:22:59.563031 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:59.563041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:59.563142 (+ 101us) server_negotiation.cc:407] Connection header received
0808 02:22:59.563325 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:59.563332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:59.563387 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:59.563528 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:59.564323 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.565771 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.566833 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.567530 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.567673 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.570490 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:59.570495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:59.570495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:59.570507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:59.572496 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.572919 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.572922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.572923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.573030 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.573258 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.573259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.573260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.573569 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:59.573730 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:59.573863 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 02:22:59.574007 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260808 02:22:59.581437 6717 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:59.581961 6717 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:59.590628 6717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Generated new TSK 0
I20260808 02:22:59.590870 6717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab.
Entry for principal oryx/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab.
I20260808 02:22:59.619243 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:46401
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:59.701316 6731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:59.701521 6731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:59.701545 6731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:59.702627 6731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:59.702659 6731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:59.702679 6731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:59.702696 6731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:22:59.704185 6731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:59.704363 6731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:59.704530 6731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:59.709374 6736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:59.709458 6731 server_base.cc:1034] running on GCE node
W20260808 02:22:59.709371 6739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:59.709569 6737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:59.709854 6731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:59.710108 6731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:59.711268 6731 hybrid_clock.cc:648] HybridClock initialized: now 1786155779711254 us; error 26 us; skew 500 ppm
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:59.713744 6731 init.cc:377] Logged in from keytab as oryx/127.30.160.1@KRBTEST.COM (short username oryx)
I20260808 02:22:59.714187 6731 webserver.cc:460] Webserver started at http://127.30.160.1:33677/ using document root <none> and password file <none>
I20260808 02:22:59.714404 6731 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:59.714470 6731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:59.714617 6731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:59.715600 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "dc966872d29b4e66897e4629b2e8e52c"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "38c3f5b22313458ac7849282bdf9fd87"
server_key_iv: "df641eee0d04812d4d244d85acc96d4b"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.716023 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "dc966872d29b4e66897e4629b2e8e52c"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "38c3f5b22313458ac7849282bdf9fd87"
server_key_iv: "df641eee0d04812d4d244d85acc96d4b"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.717343 6731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:59.718101 6746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:59.718302 6731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:59.718402 6731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "dc966872d29b4e66897e4629b2e8e52c"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "38c3f5b22313458ac7849282bdf9fd87"
server_key_iv: "df641eee0d04812d4d244d85acc96d4b"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.718487 6731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:59.743443 6731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:59.745483 6731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:59.745648 6731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:59.746178 6731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:59.746244 6731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:59.746282 6731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:59.746311 6731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:59.753996 6731 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42727
I20260808 02:22:59.754372 6731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:22:59.754362 6865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42727 every 8 connection(s)
I20260808 02:22:59.756592 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.746689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:55807 (local address 127.30.160.62:46401)
0808 02:22:59.746762 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:59.746764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:59.746997 (+ 233us) server_negotiation.cc:407] Connection header received
0808 02:22:59.747729 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:59.747730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:59.747745 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:59.747789 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:59.748761 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.749279 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.750946 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.751174 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.751229 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.753756 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:59.753758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:59.753758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:59.753768 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:59.755385 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.755767 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.755767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.755768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.755820 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.756087 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.756088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.756088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.756189 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:59.756243 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:59.756468 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 02:22:59.756531 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:22:59.756724 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.746656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.30.160.1:55807)
0808 02:22:59.746849 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:59.746898 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:59.747617 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:59.747867 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:59.747873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:59.748336 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:59.748644 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.748652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.749388 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.749389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.750854 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.750857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.751250 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.751251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.751428 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.751989 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:59.751999 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:59.753641 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:59.755459 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.755463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.755465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.755683 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.755881 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.755882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.755882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.756027 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.756248 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:59.756252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:59.756365 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 02:22:59.756438 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:22:59.756546 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260808 02:22:59.757465 6866 heartbeater.cc:347] Connected to a master server at 127.30.160.62:46401
I20260808 02:22:59.757550 6866 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:59.757722 6866 heartbeater.cc:511] Master 127.30.160.62:46401 requested a full tablet report, sending...
I20260808 02:22:59.758286 6656 ts_manager.cc:194] Registered new tserver with Master: dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727)
I20260808 02:22:59.759209 6656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:55807
I20260808 02:22:59.763988 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.758495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.30.160.1:44021)
0808 02:22:59.758657 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:59.758668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:59.758762 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:59.758903 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:59.758903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:59.759078 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:59.759150 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.759151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.759915 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.759916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.760510 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.760513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.760868 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.760869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.760918 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.761270 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:59.761274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:59.761566 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:59.763144 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.763145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.763146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.763306 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.763459 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.763459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.763459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.763502 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.763686 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:59.763686 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:59.763741 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:22:59.763789 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 02:22:59.763860 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260808 02:22:59.763988 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.758556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:44021 (local address 127.30.160.62:46401)
0808 02:22:59.758595 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:59.758597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:59.758766 (+ 169us) server_negotiation.cc:407] Connection header received
0808 02:22:59.758829 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:59.758830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:59.758838 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:59.758895 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:59.759217 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.759790 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.760580 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.760803 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.760867 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.761639 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:59.761641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:59.761641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:59.761646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:59.763070 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.763372 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.763373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.763373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.763420 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.763548 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.763548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.763549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.763634 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:59.763681 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:59.763802 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 02:22:59.763872 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:59.764560 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 6731
I20260808 02:22:59.764698 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:22:59.764907 31360 external_mini_cluster.cc:1428] Setting key 12e9df9809396fa0edaeb8a897d3d7ad
WARNING: no policy specified for oryx/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab.
Entry for principal oryx/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab.
I20260808 02:22:59.791260 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:46401
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:59.873210 6876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:59.873421 6876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:59.873447 6876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:59.874647 6876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:59.874691 6876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:59.874711 6876 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:22:59.874779 6876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:22:59.876358 6876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:59.876608 6876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:59.876852 6876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:59.881908 6882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:59.881906 6881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:59.881906 6884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:59.882072 6876 server_base.cc:1034] running on GCE node
I20260808 02:22:59.882440 6876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:59.882747 6876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:59.883953 6876 hybrid_clock.cc:648] HybridClock initialized: now 1786155779883935 us; error 47 us; skew 500 ppm
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:59.886406 6876 init.cc:377] Logged in from keytab as oryx/127.30.160.2@KRBTEST.COM (short username oryx)
I20260808 02:22:59.886915 6876 webserver.cc:460] Webserver started at http://127.30.160.2:41511/ using document root <none> and password file <none>
I20260808 02:22:59.887146 6876 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:59.887223 6876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:59.887358 6876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:59.888504 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "10560615f878403f974efb4e7d866aa8"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "6f9ccf1f1054f2a7eee197e136a38961"
server_key_iv: "9a9958a685bbfd1da45a957dbbbd47b0"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.888880 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "10560615f878403f974efb4e7d866aa8"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "6f9ccf1f1054f2a7eee197e136a38961"
server_key_iv: "9a9958a685bbfd1da45a957dbbbd47b0"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.890259 6876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:59.890990 6891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:59.891160 6876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:22:59.891259 6876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "10560615f878403f974efb4e7d866aa8"
format_stamp: "Formatted at 2026-08-08 02:22:59 on dist-test-slave-zkpd"
server_key: "6f9ccf1f1054f2a7eee197e136a38961"
server_key_iv: "9a9958a685bbfd1da45a957dbbbd47b0"
server_key_version: "encryptionkey@0"
I20260808 02:22:59.891340 6876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:59.902294 6876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:59.904157 6876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:59.904321 6876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:59.904891 6876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:22:59.904948 6876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:59.904995 6876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:22:59.905018 6876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:22:59 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:22:59.912710 6876 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:33211
I20260808 02:22:59.913113 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:33211 every 8 connection(s)
I20260808 02:22:59.913157 6876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:22:59.915073 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.905303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:35875 (local address 127.30.160.62:46401)
0808 02:22:59.905393 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:59.905396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:59.905578 (+ 182us) server_negotiation.cc:407] Connection header received
0808 02:22:59.906233 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:59.906234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:59.906248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:59.906339 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:59.907017 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.907865 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.908982 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.909336 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.909420 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.912182 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:59.912183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:59.912184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:59.912192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:59.913877 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.914233 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.914233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.914234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.914285 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.914532 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.914533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.914533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.914639 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:59.914691 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:59.914942 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 02:22:59.915007 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:22:59.915267 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.905293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.30.160.2:35875)
0808 02:22:59.905439 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:59.905480 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:59.906098 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:59.906427 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:59.906431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:59.906722 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:59.906915 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.906920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.907974 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.907974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.908906 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.908909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.909433 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.909434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.909582 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.910156 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:59.910171 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:59.912100 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:59.913948 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.913953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.913954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.914164 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.914354 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.914355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.914355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.914468 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.914697 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:59.914702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:59.914827 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:22:59.914910 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:22:59.915030 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260808 02:22:59.915802 7012 heartbeater.cc:347] Connected to a master server at 127.30.160.62:46401
I20260808 02:22:59.915918 7012 heartbeater.cc:464] Registering TS with master...
I20260808 02:22:59.916122 7012 heartbeater.cc:511] Master 127.30.160.62:46401 requested a full tablet report, sending...
I20260808 02:22:59.916183 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 6876
I20260808 02:22:59.916298 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:22:59.916543 31360 external_mini_cluster.cc:1428] Setting key 45b6e5353a7ed88dc4cbbdcb1c89a34b
I20260808 02:22:59.916544 6656 ts_manager.cc:194] Registered new tserver with Master: 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211)
I20260808 02:22:59.917022 6656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:35875
I20260808 02:22:59.923699 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.917101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:51045 (local address 127.30.160.62:46401)
0808 02:22:59.917186 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:59.917188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:59.917196 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:59.917448 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:59.917449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:59.917460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:59.917489 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:59.917926 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.918391 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.919464 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.919645 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.919701 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.920744 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:59.920745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:59.920745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:59.920751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:59.922514 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.922846 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.922848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.922848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.922907 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:59.923123 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:59.923124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:59.923125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:59.923277 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:59.923360 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:59.923535 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:22:59.923608 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:22:59.923712 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:59.916932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.30.160.2:51045)
0808 02:22:59.917053 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:59.917063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:59.917355 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:59.917547 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:59.917548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:59.917751 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:59.917830 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.917833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.918481 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.918482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.919371 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:59.919375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:59.919724 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:59.919725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:59.919785 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:59.920226 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:59.920232 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:59.920657 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:59.922584 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.922585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.922585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.922766 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.922991 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:59.922992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:59.922993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:59.923052 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:59.923353 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:59.923354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:59.923420 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 02:22:59.923492 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:22:59.923604 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab.
Entry for principal oryx/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab.
I20260808 02:22:59.942168 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:46401
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:00.025485 7022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:00.025720 7022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:00.025758 7022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:00.026902 7022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:00.026968 7022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:00.027005 7022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:23:00.027057 7022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:23:00.028640 7022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.160.62:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:00.028899 7022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:00.029131 7022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:00.034107 7028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:00.034179 7030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:00.034109 7027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:00.034396 7022 server_base.cc:1034] running on GCE node
I20260808 02:23:00.034751 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:00.035012 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:00.036157 7022 hybrid_clock.cc:648] HybridClock initialized: now 1786155780036145 us; error 27 us; skew 500 ppm
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155780, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:00.038583 7022 init.cc:377] Logged in from keytab as oryx/127.30.160.3@KRBTEST.COM (short username oryx)
I20260808 02:23:00.039011 7022 webserver.cc:460] Webserver started at http://127.30.160.3:45675/ using document root <none> and password file <none>
I20260808 02:23:00.039235 7022 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:00.039304 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:00.039459 7022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:00.040545 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
format_stamp: "Formatted at 2026-08-08 02:23:00 on dist-test-slave-zkpd"
server_key: "0e14e34c6008e1bb5add6edc8c10d3c4"
server_key_iv: "61ae3f5c6d69585b557645a279a5aea4"
server_key_version: "encryptionkey@0"
I20260808 02:23:00.040923 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
format_stamp: "Formatted at 2026-08-08 02:23:00 on dist-test-slave-zkpd"
server_key: "0e14e34c6008e1bb5add6edc8c10d3c4"
server_key_iv: "61ae3f5c6d69585b557645a279a5aea4"
server_key_version: "encryptionkey@0"
I20260808 02:23:00.042251 7022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:23:00.042977 7037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.043160 7022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:00.043268 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
format_stamp: "Formatted at 2026-08-08 02:23:00 on dist-test-slave-zkpd"
server_key: "0e14e34c6008e1bb5add6edc8c10d3c4"
server_key_iv: "61ae3f5c6d69585b557645a279a5aea4"
server_key_version: "encryptionkey@0"
I20260808 02:23:00.043345 7022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:00.062433 7022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:00.064344 7022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:00.064504 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:00.065071 7022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:00.065129 7022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.065172 7022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:00.065200 7022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155780, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:23:00.072368 7022 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:39717
I20260808 02:23:00.072453 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:39717 every 8 connection(s)
I20260808 02:23:00.072767 7022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:23:00.075052 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.065472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:43903 (local address 127.30.160.62:46401)
0808 02:23:00.065531 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.065535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.065735 (+ 200us) server_negotiation.cc:407] Connection header received
0808 02:23:00.066209 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.066210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.066225 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.066320 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.066978 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.067789 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.069494 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.069700 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.069794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.072157 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:00.072159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:00.072159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:00.072167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:00.073776 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.074127 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.074128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.074128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.074210 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.074455 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.074456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.074457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.074597 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:00.074661 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.074920 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.074987 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:23:00.075186 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.065456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.30.160.3:43903)
0808 02:23:00.065601 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.065640 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.066116 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.066363 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.066367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.066592 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:00.066878 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.066883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.067920 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.067921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.069402 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.069405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.069827 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.069828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.069984 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.070505 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:00.070514 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:00.072037 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:00.073849 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:00.073853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:00.073855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:00.074059 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:00.074283 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:00.074284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:00.074284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:00.074389 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:00.074666 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:00.074670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:00.074795 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.074883 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.074998 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260808 02:23:00.075665 7156 heartbeater.cc:347] Connected to a master server at 127.30.160.62:46401
I20260808 02:23:00.075764 7156 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:00.075986 7156 heartbeater.cc:511] Master 127.30.160.62:46401 requested a full tablet report, sending...
I20260808 02:23:00.076406 6657 ts_manager.cc:194] Registered new tserver with Master: 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.076900 6657 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:43903
I20260808 02:23:00.077080 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7022
I20260808 02:23:00.077189 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:23:00.077419 31360 external_mini_cluster.cc:1428] Setting key 243ec9664a22cb9170f744f6a63af9ee
I20260808 02:23:00.078267 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:23:00.083269 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.076744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:36273 (local address 127.30.160.62:46401)
0808 02:23:00.076781 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.076783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.076909 (+ 126us) server_negotiation.cc:407] Connection header received
0808 02:23:00.077654 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.077655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.077667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.077724 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.078122 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.078668 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.079346 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.079554 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.079629 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.080680 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:00.080681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:00.080681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:00.080687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:00.082107 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.082557 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.082558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.082559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.082624 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.082840 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.082840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.082841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.082933 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:00.082984 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.083163 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.083215 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:23:00.083382 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.076678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.30.160.3:36273)
0808 02:23:00.076843 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.076853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.077572 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.077769 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.077770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.077971 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:00.078032 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.078034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.078769 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.078770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.079263 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.079265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.079628 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.079629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.079702 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.080136 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:00.080139 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:00.080590 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:00.082196 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:00.082197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:00.082198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:00.082472 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:00.082699 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:00.082700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:00.082700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:00.082775 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:00.083002 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:00.083003 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:00.083063 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.083149 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.083258 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:23:00.100698 6727 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:00.095568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.30.160.62:46401)
0808 02:23:00.095624 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.095626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.095641 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:23:00.095662 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.095662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.095671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.095718 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.096339 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.096790 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.097498 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.097666 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.097714 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.100585 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39480: BlockingRecv error: recv got EOF from 127.0.0.1:39480 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:23:00.107772 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.101578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.30.160.62:46401)
0808 02:23:00.101629 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.101631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.101726 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:23:00.101762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.101762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.101770 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.101821 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.102317 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.102740 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.103366 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.103530 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.103580 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.105573 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:00.105575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:00.105575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:00.105581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:00.106943 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.107228 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.107229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.107229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.107279 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.107423 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.107424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.107424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.107511 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:00.107579 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.107659 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.107712 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:23:00.109493 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:23:00.111064 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:23:00.117754 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.113328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33211 (local address 127.0.0.1:40484)
0808 02:23:00.113804 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.113814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.113885 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.114152 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.114153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.114155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.114205 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.114208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.115471 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.115472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.116605 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.116607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.117438 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.117439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.117528 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.117542 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.117618 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.117669 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":72,"spinlock_wait_cycles":896,"thread_start_us":97,"threads_started":1}
I20260808 02:23:00.117763 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.113605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.30.160.2:33211)
0808 02:23:00.113923 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.113927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.113935 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:23:00.113979 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.113981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.113997 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.114078 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.114286 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.115368 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.116693 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.117350 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.117435 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.117515 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.117633 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.117670 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":106,"threads_started":1}
I20260808 02:23:00.117830 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.113605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.30.160.1:42727)
0808 02:23:00.113803 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.113808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.113827 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:23:00.113929 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.113932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.113951 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.114036 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.114278 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.115105 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.116519 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.117263 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.117311 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.117385 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.117657 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.117696 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":109,"threads_started":1}
I20260808 02:23:00.117947 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.113432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.30.160.3:39717)
0808 02:23:00.113618 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.113624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.114054 (+ 430us) server_negotiation.cc:407] Connection header received
0808 02:23:00.114183 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.114185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.114202 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.114255 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.114484 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.115504 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.116783 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.117483 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.117547 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.117617 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.117796 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.117841 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260808 02:23:00.118031 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.113327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42727 (local address 127.0.0.1:54008)
0808 02:23:00.113638 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.113691 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.113795 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.114119 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.114123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.114125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.114189 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.114191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.115183 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.115184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.116422 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.116425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.117451 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.117452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.117549 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.117553 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.117903 (+ 350us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.117949 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":96,"threads_started":1}
I20260808 02:23:00.118155 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.113334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:39717 (local address 127.0.0.1:42354)
0808 02:23:00.113990 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.113999 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.114084 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.114365 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.114366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.114368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.114401 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.114402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.115589 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.115590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.116623 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.116626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.117648 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.117649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.117703 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.117706 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.118054 (+ 348us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.118097 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260808 02:23:00.118899 7089 tablet_service.cc:1467] Processing CreateTablet for tablet f80513eaea22471493ddacba50476edc (DEFAULT_TABLE table=test-table [id=665bf5ee56e1462cbd31ecb8d85a2b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.119019 6944 tablet_service.cc:1467] Processing CreateTablet for tablet f80513eaea22471493ddacba50476edc (DEFAULT_TABLE table=test-table [id=665bf5ee56e1462cbd31ecb8d85a2b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.119020 6797 tablet_service.cc:1467] Processing CreateTablet for tablet f80513eaea22471493ddacba50476edc (DEFAULT_TABLE table=test-table [id=665bf5ee56e1462cbd31ecb8d85a2b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.119307 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80513eaea22471493ddacba50476edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.119318 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80513eaea22471493ddacba50476edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.119408 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80513eaea22471493ddacba50476edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.122792 7186 tablet_bootstrap.cc:492] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Bootstrap starting.
I20260808 02:23:00.123049 7187 tablet_bootstrap.cc:492] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Bootstrap starting.
I20260808 02:23:00.123190 7188 tablet_bootstrap.cc:492] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Bootstrap starting.
I20260808 02:23:00.123657 7186 tablet_bootstrap.cc:654] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.123754 7188 tablet_bootstrap.cc:654] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.123781 7187 tablet_bootstrap.cc:654] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.124076 7186 log.cc:824] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:00.124099 7188 log.cc:824] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:00.124159 7187 log.cc:824] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:00.124848 7186 tablet_bootstrap.cc:492] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: No bootstrap required, opened a new log
I20260808 02:23:00.124905 7186 ts_tablet_manager.cc:1397] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:23:00.125057 7188 tablet_bootstrap.cc:492] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: No bootstrap required, opened a new log
I20260808 02:23:00.125128 7188 ts_tablet_manager.cc:1397] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:23:00.126428 7186 raft_consensus.cc:348] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.126564 7186 raft_consensus.cc:374] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.126587 7186 raft_consensus.cc:729] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc966872d29b4e66897e4629b2e8e52c, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.126520 7188 raft_consensus.cc:348] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.126698 7188 raft_consensus.cc:374] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.126710 7186 consensus_queue.cc:260] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.126750 7188 raft_consensus.cc:729] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.126871 7188 consensus_queue.cc:260] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.127033 6866 heartbeater.cc:503] Master 127.30.160.62:46401 was elected leader, sending a full tablet report...
I20260808 02:23:00.127015 7186 ts_tablet_manager.cc:1428] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:23:00.127161 7188 ts_tablet_manager.cc:1428] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:23:00.127350 7012 heartbeater.cc:503] Master 127.30.160.62:46401 was elected leader, sending a full tablet report...
I20260808 02:23:00.127386 7187 tablet_bootstrap.cc:492] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: No bootstrap required, opened a new log
I20260808 02:23:00.127444 7187 ts_tablet_manager.cc:1397] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260808 02:23:00.129235 7187 raft_consensus.cc:348] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.129392 7187 raft_consensus.cc:374] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.129432 7187 raft_consensus.cc:729] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0d0ce718c43959fc4fdaaa752bfac, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.129550 7187 consensus_queue.cc:260] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.129787 7187 ts_tablet_manager.cc:1428] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260808 02:23:00.129859 7156 heartbeater.cc:503] Master 127.30.160.62:46401 was elected leader, sending a full tablet report...
I20260808 02:23:00.150210 7194 raft_consensus.cc:482] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:23:00.150339 7194 raft_consensus.cc:504] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.150647 7194 leader_election.cc:288] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717), dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727)
I20260808 02:23:00.154636 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.150936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:49693 (local address 127.30.160.1:42727)
0808 02:23:00.150990 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.150993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.151024 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:23:00.151059 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.151060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.151067 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.151131 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.151304 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.152101 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.153666 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.154252 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.154336 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.154344 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.154522 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.154560 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:23:00.154636 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.150927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:42397 (local address 127.30.160.3:39717)
0808 02:23:00.150991 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.150993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.151125 (+ 132us) server_negotiation.cc:407] Connection header received
0808 02:23:00.151192 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.151195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.151208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.151251 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.151405 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.152124 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.153666 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.154251 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.154333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.154343 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.154522 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.154560 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:23:00.154649 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.150850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42727 (local address 127.30.160.2:49693)
0808 02:23:00.150912 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.150925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.150971 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.151148 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.151149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.151151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.151226 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.151228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.152199 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.152200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.153564 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.153569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.154347 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.154348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.154433 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.154440 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.154499 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.154555 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":106}
I20260808 02:23:00.154670 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.150850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:39717 (local address 127.30.160.2:42397)
0808 02:23:00.151048 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.151065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.151139 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.151254 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.151255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.151256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.151329 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.151330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.152199 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.152200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.153564 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.153568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.154347 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.154348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.154433 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.154440 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.154499 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.154555 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":64,"thread_start_us":113,"threads_started":1}
I20260808 02:23:00.155063 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80513eaea22471493ddacba50476edc" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c" is_pre_election: true
I20260808 02:23:00.155241 6817 raft_consensus.cc:2454] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 0.
I20260808 02:23:00.155252 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80513eaea22471493ddacba50476edc" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" is_pre_election: true
I20260808 02:23:00.155440 7110 raft_consensus.cc:2454] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 0.
I20260808 02:23:00.155544 6895 leader_election.cc:302] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, dc966872d29b4e66897e4629b2e8e52c; no voters:
I20260808 02:23:00.155660 7194 raft_consensus.cc:2781] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:23:00.155743 7194 raft_consensus.cc:482] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:23:00.155776 7194 raft_consensus.cc:3037] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.157122 7194 raft_consensus.cc:504] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.157275 7194 leader_election.cc:288] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Requested vote from peers 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717), dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727)
I20260808 02:23:00.157512 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80513eaea22471493ddacba50476edc" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
I20260808 02:23:00.157605 7110 raft_consensus.cc:3037] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.157663 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80513eaea22471493ddacba50476edc" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c"
I20260808 02:23:00.157770 6817 raft_consensus.cc:3037] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.158588 7110 raft_consensus.cc:2454] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 1.
I20260808 02:23:00.158686 6817 raft_consensus.cc:2454] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 1.
I20260808 02:23:00.158872 6895 leader_election.cc:302] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, dc966872d29b4e66897e4629b2e8e52c; no voters:
I20260808 02:23:00.159276 7194 raft_consensus.cc:2781] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:00.159459 7194 raft_consensus.cc:686] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Running, Role: LEADER
I20260808 02:23:00.159569 7194 consensus_queue.cc:237] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.160499 6657 catalog_manager.cc:5495] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10560615f878403f974efb4e7d866aa8 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } health_report { overall_health: HEALTHY } } }
W20260808 02:23:00.163672 7014 tablet.cc:2369] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:23:00.175486 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.173011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.30.160.2:33211)
0808 02:23:00.173075 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.173078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.173088 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:00.173165 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.173166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.173185 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.173271 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.173433 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.173912 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.174670 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.174876 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.174949 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.175134 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.175235 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.175332 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.175383 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.175404 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:23:00.177356 6817 raft_consensus.cc:1261] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Refusing update from remote peer 10560615f878403f974efb4e7d866aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.177366 7110 raft_consensus.cc:1261] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Refusing update from remote peer 10560615f878403f974efb4e7d866aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.177685 7194 consensus_queue.cc:1035] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.177870 7197 consensus_queue.cc:1035] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.180614 7201 mvcc.cc:205] Tried to move back new op lower bound from 7316094075603783680 to 7316094075534442496. Current Snapshot: MvccSnapshot[applied={T|T < 7316094075603783680}]
I20260808 02:23:00.182089 7202 mvcc.cc:205] Tried to move back new op lower bound from 7316094075603783680 to 7316094075534442496. Current Snapshot: MvccSnapshot[applied={T|T < 7316094075603783680}]
I20260808 02:23:00.182471 6657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.182570 6657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.183328 6657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Sending DeleteTablet for 3 replicas of tablet f80513eaea22471493ddacba50476edc
I20260808 02:23:00.183663 6944 tablet_service.cc:1514] Processing DeleteTablet for tablet f80513eaea22471493ddacba50476edc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:40484
I20260808 02:23:00.183708 7089 tablet_service.cc:1514] Processing DeleteTablet for tablet f80513eaea22471493ddacba50476edc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42354
I20260808 02:23:00.183713 6797 tablet_service.cc:1514] Processing DeleteTablet for tablet f80513eaea22471493ddacba50476edc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:54008
I20260808 02:23:00.184006 7211 tablet_replica.cc:330] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: stopping tablet replica
I20260808 02:23:00.184227 7211 raft_consensus.cc:2229] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.184216 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:23:00.184352 7211 raft_consensus.cc:2258] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:23:00.184383 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:23:00.184249 7212 tablet_replica.cc:330] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: stopping tablet replica
I20260808 02:23:00.184621 7212 raft_consensus.cc:2229] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.184749 7212 raft_consensus.cc:2258] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.185192 7212 ts_tablet_manager.cc:1905] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.185748 6944 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcb1ee264314d1aadcb580c0a886b8c (DEFAULT_TABLE table=test-table [id=9c3cafe1066f4371861320b0c89094d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.185879 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcb1ee264314d1aadcb580c0a886b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.185931 6797 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcb1ee264314d1aadcb580c0a886b8c (DEFAULT_TABLE table=test-table [id=9c3cafe1066f4371861320b0c89094d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.186029 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcb1ee264314d1aadcb580c0a886b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.186012 7089 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcb1ee264314d1aadcb580c0a886b8c (DEFAULT_TABLE table=test-table [id=9c3cafe1066f4371861320b0c89094d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.186128 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcb1ee264314d1aadcb580c0a886b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.187103 7212 ts_tablet_manager.cc:1918] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:00.187168 7212 log.cc:1196] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/f80513eaea22471493ddacba50476edc
I20260808 02:23:00.187249 7188 tablet_bootstrap.cc:492] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Bootstrap starting.
I20260808 02:23:00.187433 7186 tablet_bootstrap.cc:492] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Bootstrap starting.
I20260808 02:23:00.187444 7212 ts_tablet_manager.cc:1939] T f80513eaea22471493ddacba50476edc P dc966872d29b4e66897e4629b2e8e52c: Deleting consensus metadata
I20260808 02:23:00.187700 7188 tablet_bootstrap.cc:654] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.187723 7187 tablet_bootstrap.cc:492] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Bootstrap starting.
I20260808 02:23:00.188084 6643 catalog_manager.cc:4841] TS dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727): tablet f80513eaea22471493ddacba50476edc (table test-table [id=665bf5ee56e1462cbd31ecb8d85a2b48]) successfully deleted
I20260808 02:23:00.188179 7186 tablet_bootstrap.cc:654] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.188203 7211 ts_tablet_manager.cc:1905] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.188405 7187 tablet_bootstrap.cc:654] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.188782 7188 tablet_bootstrap.cc:492] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: No bootstrap required, opened a new log
I20260808 02:23:00.188848 7188 ts_tablet_manager.cc:1397] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:23:00.188987 7188 raft_consensus.cc:348] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.189059 7188 raft_consensus.cc:374] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.189081 7188 raft_consensus.cc:729] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.189121 7188 consensus_queue.cc:260] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.189226 7188 ts_tablet_manager.cc:1428] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.189605 7186 tablet_bootstrap.cc:492] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: No bootstrap required, opened a new log
I20260808 02:23:00.189656 7186 ts_tablet_manager.cc:1397] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:23:00.189734 7210 tablet_replica.cc:330] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: stopping tablet replica
I20260808 02:23:00.189831 7210 raft_consensus.cc:2229] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:23:00.189791 7186 raft_consensus.cc:348] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.189867 7186 raft_consensus.cc:374] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.189888 7186 raft_consensus.cc:729] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc966872d29b4e66897e4629b2e8e52c, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.189927 7210 raft_consensus.cc:2258] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.189929 7186 consensus_queue.cc:260] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.190007 7186 ts_tablet_manager.cc:1428] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:23:00.190490 7211 ts_tablet_manager.cc:1918] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:00.190567 7211 log.cc:1196] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/f80513eaea22471493ddacba50476edc
I20260808 02:23:00.190685 7210 ts_tablet_manager.cc:1905] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.190860 7211 ts_tablet_manager.cc:1939] T f80513eaea22471493ddacba50476edc P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting consensus metadata
I20260808 02:23:00.191262 6640 catalog_manager.cc:4841] TS 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717): tablet f80513eaea22471493ddacba50476edc (table test-table [id=665bf5ee56e1462cbd31ecb8d85a2b48]) successfully deleted
I20260808 02:23:00.191671 7187 tablet_bootstrap.cc:492] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: No bootstrap required, opened a new log
I20260808 02:23:00.191732 7187 ts_tablet_manager.cc:1397] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 02:23:00.191872 7187 raft_consensus.cc:348] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.191975 7187 raft_consensus.cc:374] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.192016 7187 raft_consensus.cc:729] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0d0ce718c43959fc4fdaaa752bfac, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.192116 7187 consensus_queue.cc:260] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.192238 7210 ts_tablet_manager.cc:1918] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:00.192309 7210 log.cc:1196] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/f80513eaea22471493ddacba50476edc
I20260808 02:23:00.192427 7187 ts_tablet_manager.cc:1428] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:00.192557 7210 ts_tablet_manager.cc:1939] T f80513eaea22471493ddacba50476edc P 10560615f878403f974efb4e7d866aa8: Deleting consensus metadata
I20260808 02:23:00.193127 6642 catalog_manager.cc:4841] TS 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211): tablet f80513eaea22471493ddacba50476edc (table test-table [id=665bf5ee56e1462cbd31ecb8d85a2b48]) successfully deleted
W20260808 02:23:00.255189 6867 tablet.cc:2369] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:23:00.324168 7157 tablet.cc:2369] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:23:00.324759 7193 raft_consensus.cc:482] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:23:00.324882 7193 raft_consensus.cc:504] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.325332 7193 leader_election.cc:288] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211), 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.328560 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.325637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:60077 (local address 127.30.160.3:39717)
0808 02:23:00.325715 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.325717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.325809 (+ 92us) server_negotiation.cc:407] Connection header received
0808 02:23:00.325875 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.325877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.325889 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.325943 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.326126 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.326703 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.327829 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.328237 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.328298 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.328305 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.328478 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.328506 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:23:00.328593 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.325553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:39717 (local address 127.30.160.1:60077)
0808 02:23:00.325707 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.325720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.325804 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.325962 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.325963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.325966 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.326052 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.326054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.326787 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.326788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.327667 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.327669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.328322 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.328323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.328401 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.328405 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.328467 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.328509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260808 02:23:00.328873 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb1ee264314d1aadcb580c0a886b8c" candidate_uuid: "dc966872d29b4e66897e4629b2e8e52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" is_pre_election: true
I20260808 02:23:00.328979 7110 raft_consensus.cc:2454] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc966872d29b4e66897e4629b2e8e52c in term 0.
I20260808 02:23:00.329025 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.325553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33211 (local address 127.30.160.1:57201)
0808 02:23:00.325833 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.325843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.325905 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.326315 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.326316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.326319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.326381 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.326383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.327078 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.327079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.327829 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.327831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.328768 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.328770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.328831 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.328834 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.328894 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.328931 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":20,"thread_start_us":55,"threads_started":1}
I20260808 02:23:00.329027 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.325636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:57201 (local address 127.30.160.2:33211)
0808 02:23:00.325700 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.325703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.326097 (+ 394us) server_negotiation.cc:407] Connection header received
0808 02:23:00.326131 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.326132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.326216 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.326261 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.326446 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.326991 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.328122 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.328655 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.328728 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.328736 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.328907 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.328957 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:23:00.329167 6747 leader_election.cc:302] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa0d0ce718c43959fc4fdaaa752bfac, dc966872d29b4e66897e4629b2e8e52c; no voters:
I20260808 02:23:00.329301 7193 raft_consensus.cc:2781] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:23:00.329353 7193 raft_consensus.cc:482] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:23:00.329376 7193 raft_consensus.cc:3037] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.329550 6964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb1ee264314d1aadcb580c0a886b8c" candidate_uuid: "dc966872d29b4e66897e4629b2e8e52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10560615f878403f974efb4e7d866aa8" is_pre_election: true
I20260808 02:23:00.329694 6964 raft_consensus.cc:2454] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc966872d29b4e66897e4629b2e8e52c in term 0.
I20260808 02:23:00.330241 7193 raft_consensus.cc:504] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.330391 7193 leader_election.cc:288] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [CANDIDATE]: Term 1 election: Requested vote from peers 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211), 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.330539 6964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb1ee264314d1aadcb580c0a886b8c" candidate_uuid: "dc966872d29b4e66897e4629b2e8e52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10560615f878403f974efb4e7d866aa8"
I20260808 02:23:00.330576 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb1ee264314d1aadcb580c0a886b8c" candidate_uuid: "dc966872d29b4e66897e4629b2e8e52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
I20260808 02:23:00.330652 7110 raft_consensus.cc:3037] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.330657 6964 raft_consensus.cc:3037] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.331252 7110 raft_consensus.cc:2454] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc966872d29b4e66897e4629b2e8e52c in term 1.
I20260808 02:23:00.331388 6964 raft_consensus.cc:2454] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc966872d29b4e66897e4629b2e8e52c in term 1.
I20260808 02:23:00.331423 6747 leader_election.cc:302] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa0d0ce718c43959fc4fdaaa752bfac, dc966872d29b4e66897e4629b2e8e52c; no voters:
I20260808 02:23:00.331535 7193 raft_consensus.cc:2781] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:00.331693 7193 raft_consensus.cc:686] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 1 LEADER]: Becoming Leader. State: Replica: dc966872d29b4e66897e4629b2e8e52c, State: Running, Role: LEADER
I20260808 02:23:00.331791 7193 consensus_queue.cc:237] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.332520 6657 catalog_manager.cc:5495] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c reported cstate change: term changed from 0 to 1, leader changed from <none> to dc966872d29b4e66897e4629b2e8e52c (127.30.160.1). New cstate: current_term: 1 leader_uuid: "dc966872d29b4e66897e4629b2e8e52c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155779, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.62@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:23:00.395453 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.388604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:46401 (local address 127.0.0.1:48754)
0808 02:23:00.388801 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.388810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.388896 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.389060 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.389061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.389399 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:00.389445 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.389447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.390142 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.390143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.390648 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.390649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.391078 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.391079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.391131 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.391585 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:00.391589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:00.392955 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:00.394430 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:00.394436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:00.394438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:00.394683 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:00.394901 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:00.394902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:00.394902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:00.394961 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:00.395174 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:00.395180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:00.395221 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.395271 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.395357 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":118,"threads_started":1}
I20260808 02:23:00.395495 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.388670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.30.160.62:46401)
0808 02:23:00.388755 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.388758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.388869 (+ 111us) server_negotiation.cc:407] Connection header received
0808 02:23:00.388968 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.388969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.388979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.389055 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.389528 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.390061 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.390749 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.391003 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.391080 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.393039 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:00.393040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:00.393040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:00.393047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:00.394341 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.394756 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.394757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.394757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.394839 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.395010 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.395011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.395011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.395125 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:00.395171 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.395311 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.395401 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":41}
I20260808 02:23:00.397403 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:48754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:23:00.397622 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:23:00.399086 6797 tablet_service.cc:1467] Processing CreateTablet for tablet 17a74dc62a434451832ed59299e62c02 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5b3b03ef5794a77bd013f9b669032a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:23:00.399083 6944 tablet_service.cc:1467] Processing CreateTablet for tablet 17a74dc62a434451832ed59299e62c02 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5b3b03ef5794a77bd013f9b669032a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:23:00.399082 7089 tablet_service.cc:1467] Processing CreateTablet for tablet 17a74dc62a434451832ed59299e62c02 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5b3b03ef5794a77bd013f9b669032a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:23:00.399276 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a74dc62a434451832ed59299e62c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.399278 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a74dc62a434451832ed59299e62c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.399277 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a74dc62a434451832ed59299e62c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.400709 7186 tablet_bootstrap.cc:492] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: Bootstrap starting.
I20260808 02:23:00.400857 7187 tablet_bootstrap.cc:492] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: Bootstrap starting.
I20260808 02:23:00.400897 7188 tablet_bootstrap.cc:492] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: Bootstrap starting.
I20260808 02:23:00.401134 7186 tablet_bootstrap.cc:654] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.401247 7188 tablet_bootstrap.cc:654] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.401289 7187 tablet_bootstrap.cc:654] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.401988 7187 tablet_bootstrap.cc:492] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: No bootstrap required, opened a new log
I20260808 02:23:00.401995 7188 tablet_bootstrap.cc:492] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: No bootstrap required, opened a new log
I20260808 02:23:00.402043 7187 ts_tablet_manager.cc:1397] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:00.402061 7188 ts_tablet_manager.cc:1397] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:23:00.402189 7188 raft_consensus.cc:348] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.402252 7188 raft_consensus.cc:374] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.402212 7187 raft_consensus.cc:348] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.402275 7188 raft_consensus.cc:729] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.402283 7187 raft_consensus.cc:374] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.402315 7187 raft_consensus.cc:729] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0d0ce718c43959fc4fdaaa752bfac, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.402328 7188 consensus_queue.cc:260] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.402360 7187 consensus_queue.cc:260] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.402438 7188 ts_tablet_manager.cc:1428] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.402467 7187 ts_tablet_manager.cc:1428] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.402457 7207 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.402537 7207 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.402648 7186 tablet_bootstrap.cc:492] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: No bootstrap required, opened a new log
I20260808 02:23:00.402607 7195 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.402693 7186 ts_tablet_manager.cc:1397] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:23:00.402691 7195 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.402832 7186 raft_consensus.cc:348] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.402904 7186 raft_consensus.cc:374] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.402925 7186 raft_consensus.cc:729] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc966872d29b4e66897e4629b2e8e52c, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.402958 7186 consensus_queue.cc:260] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.403046 7186 ts_tablet_manager.cc:1428] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.403098 7193 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.403191 7193 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.557979 7207 raft_consensus.cc:482] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:23:00.558104 7207 raft_consensus.cc:504] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.558322 7207 leader_election.cc:288] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727), 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.558550 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a74dc62a434451832ed59299e62c02" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c" is_pre_election: true
I20260808 02:23:00.558600 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a74dc62a434451832ed59299e62c02" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" is_pre_election: true
I20260808 02:23:00.558678 6817 raft_consensus.cc:2454] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 0.
I20260808 02:23:00.558683 7110 raft_consensus.cc:2454] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 0.
I20260808 02:23:00.558854 6892 leader_election.cc:302] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, 6fa0d0ce718c43959fc4fdaaa752bfac; no voters:
I20260808 02:23:00.559000 7207 raft_consensus.cc:2781] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:23:00.559077 7207 raft_consensus.cc:482] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:23:00.559108 7207 raft_consensus.cc:3037] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.559859 7207 raft_consensus.cc:504] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.560057 7207 leader_election.cc:288] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Requested vote from peers dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727), 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.560233 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a74dc62a434451832ed59299e62c02" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c"
I20260808 02:23:00.560236 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a74dc62a434451832ed59299e62c02" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
I20260808 02:23:00.560310 7110 raft_consensus.cc:3037] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.560333 6817 raft_consensus.cc:3037] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.561003 7110 raft_consensus.cc:2454] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 1.
I20260808 02:23:00.561084 6817 raft_consensus.cc:2454] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 1.
I20260808 02:23:00.561232 6892 leader_election.cc:302] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, 6fa0d0ce718c43959fc4fdaaa752bfac; no voters:
I20260808 02:23:00.561347 7207 raft_consensus.cc:2781] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:00.561424 7207 raft_consensus.cc:686] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Running, Role: LEADER
I20260808 02:23:00.561486 7207 consensus_queue.cc:237] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } }
I20260808 02:23:00.561785 7197 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: TxnStatusTablet state changed. Reason: New leader 10560615f878403f974efb4e7d866aa8. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.561890 7197 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:23:00.562059 7231 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:23:00.562180 7231 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:23:00.562145 6656 catalog_manager.cc:5495] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10560615f878403f974efb4e7d866aa8 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } health_report { overall_health: HEALTHY } } }
I20260808 02:23:00.585031 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.582514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.30.160.2:33211)
0808 02:23:00.582564 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.582566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.582576 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:00.582670 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.582671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.582682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.582757 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.582921 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.583417 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.584278 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.584480 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.584556 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.584741 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.584784 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.584850 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.584924 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.584947 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:23:00.585039 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.582449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33211 (local address 127.0.0.1:37030)
0808 02:23:00.582521 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.582534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.582591 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.582768 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.582770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.582771 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:23:00.582830 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.582833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.583526 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.583527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.584189 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.584192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.584569 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.584570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.584642 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.584657 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.584853 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.584865 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.584919 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.584970 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:23:00.586102 6817 raft_consensus.cc:1261] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Refusing update from remote peer 10560615f878403f974efb4e7d866aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.586180 7110 raft_consensus.cc:1261] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Refusing update from remote peer 10560615f878403f974efb4e7d866aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.586346 7207 consensus_queue.cc:1035] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.586464 7197 consensus_queue.cc:1035] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.587508 7207 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.587589 7207 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:23:00.587656 7200 mvcc.cc:205] Tried to move back new op lower bound from 7316094077278937088 to 7316094077180256256. Current Snapshot: MvccSnapshot[applied={T|T < 7316094077278937088}]
I20260808 02:23:00.587719 7219 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.587831 7219 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.588451 7193 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: TxnStatusTablet state changed. Reason: New leader 10560615f878403f974efb4e7d866aa8. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.588526 7193 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P dc966872d29b4e66897e4629b2e8e52c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.588815 7195 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: TxnStatusTablet state changed. Reason: New leader 10560615f878403f974efb4e7d866aa8. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.588894 7195 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.589205 7194 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.589303 7194 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 10560615f878403f974efb4e7d866aa8: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:23:00.589730 7195 tablet_replica.cc:439] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } }
I20260808 02:23:00.589807 7195 tablet_replica.cc:442] T 17a74dc62a434451832ed59299e62c02 P 6fa0d0ce718c43959fc4fdaaa752bfac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:23:00.592140 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.589352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.30.160.1:42727)
0808 02:23:00.589392 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.589394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.589401 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:23:00.589435 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.589436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.589456 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.589519 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.589706 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.590277 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.591238 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.591508 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.591589 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.591766 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.591881 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.591987 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.592043 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.592065 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:23:00.597229 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.594734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:45559 (local address 127.30.160.2:33211)
0808 02:23:00.594788 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.594790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.594944 (+ 154us) server_negotiation.cc:407] Connection header received
0808 02:23:00.595011 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.595012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.595023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.595093 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.595225 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.595671 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.596510 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.596694 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.596775 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.596985 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.597020 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.597081 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.597142 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.597162 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:23:00.597260 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.594675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33211 (local address 127.30.160.1:45559)
0808 02:23:00.594872 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.594886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.594959 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.595094 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.595095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.595097 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:23:00.595153 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.595154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.595754 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.595756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.596414 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.596416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.596793 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.596794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.596874 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.596888 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.597083 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.597085 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.597138 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.597179 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260808 02:23:00.599696 7110 raft_consensus.cc:1261] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Refusing update from remote peer dc966872d29b4e66897e4629b2e8e52c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.599802 6964 raft_consensus.cc:1261] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Refusing update from remote peer dc966872d29b4e66897e4629b2e8e52c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.600112 7193 consensus_queue.cc:1035] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.600232 7219 consensus_queue.cc:1035] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.608911 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.605989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:56545 (local address 127.30.160.1:42727)
0808 02:23:00.606024 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.606025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.606215 (+ 190us) server_negotiation.cc:407] Connection header received
0808 02:23:00.606318 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.606319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.606333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.606396 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.606531 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.607001 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.608218 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.608393 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.608441 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.608677 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.608712 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.608764 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.608828 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.608846 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 02:23:00.608944 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.605923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42727 (local address 127.30.160.2:56545)
0808 02:23:00.606154 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.606164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.606240 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.606401 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.606402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.606403 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:23:00.606448 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.606450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.607079 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.607080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.608125 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.608129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.608462 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.608463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.608525 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.608562 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.608767 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.608771 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.608819 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.608866 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260808 02:23:00.618664 6657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.618768 6657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.619379 6657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Sending DeleteTablet for 3 replicas of tablet 0dcb1ee264314d1aadcb580c0a886b8c
I20260808 02:23:00.619686 7089 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcb1ee264314d1aadcb580c0a886b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42354
I20260808 02:23:00.619789 7211 tablet_replica.cc:330] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: stopping tablet replica
I20260808 02:23:00.619843 7211 raft_consensus.cc:2229] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.619880 7211 raft_consensus.cc:2258] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.620026 6944 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcb1ee264314d1aadcb580c0a886b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:40484
I20260808 02:23:00.620111 7210 tablet_replica.cc:330] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: stopping tablet replica
I20260808 02:23:00.620157 7210 raft_consensus.cc:2229] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.620195 7210 raft_consensus.cc:2258] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.620549 7210 ts_tablet_manager.cc:1905] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.622135 7210 ts_tablet_manager.cc:1918] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:23:00.622197 7210 log.cc:1196] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/0dcb1ee264314d1aadcb580c0a886b8c
I20260808 02:23:00.622422 7210 ts_tablet_manager.cc:1939] T 0dcb1ee264314d1aadcb580c0a886b8c P 10560615f878403f974efb4e7d866aa8: Deleting consensus metadata
I20260808 02:23:00.622737 6797 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcb1ee264314d1aadcb580c0a886b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:54008
I20260808 02:23:00.622822 7212 tablet_replica.cc:330] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: stopping tablet replica
I20260808 02:23:00.622789 6642 catalog_manager.cc:4841] TS 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211): tablet 0dcb1ee264314d1aadcb580c0a886b8c (table test-table [id=9c3cafe1066f4371861320b0c89094d2]) successfully deleted
I20260808 02:23:00.622861 7212 raft_consensus.cc:2229] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:23:00.622936 7212 raft_consensus.cc:2258] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.623368 7212 ts_tablet_manager.cc:1905] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.624917 7212 ts_tablet_manager.cc:1918] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:23:00.624979 7212 log.cc:1196] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/0dcb1ee264314d1aadcb580c0a886b8c
I20260808 02:23:00.625181 7212 ts_tablet_manager.cc:1939] T 0dcb1ee264314d1aadcb580c0a886b8c P dc966872d29b4e66897e4629b2e8e52c: Deleting consensus metadata
I20260808 02:23:00.625499 6643 catalog_manager.cc:4841] TS dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727): tablet 0dcb1ee264314d1aadcb580c0a886b8c (table test-table [id=9c3cafe1066f4371861320b0c89094d2]) successfully deleted
I20260808 02:23:00.628624 7211 ts_tablet_manager.cc:1905] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.630146 7211 ts_tablet_manager.cc:1918] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:23:00.630208 7211 log.cc:1196] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/0dcb1ee264314d1aadcb580c0a886b8c
I20260808 02:23:00.630416 7211 ts_tablet_manager.cc:1939] T 0dcb1ee264314d1aadcb580c0a886b8c P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting consensus metadata
I20260808 02:23:00.630731 6640 catalog_manager.cc:4841] TS 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717): tablet 0dcb1ee264314d1aadcb580c0a886b8c (table test-table [id=9c3cafe1066f4371861320b0c89094d2]) successfully deleted
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:23:00.657249 6727 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:00.650080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.30.160.62:46401)
0808 02:23:00.650132 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.650134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.650263 (+ 129us) server_negotiation.cc:407] Connection header received
0808 02:23:00.650406 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.650407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.650420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.650453 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.651243 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.652057 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.653159 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.653459 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.653508 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.657183 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48764: BlockingRecv error: recv got EOF from 127.0.0.1:48764 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM
I20260808 02:23:00.667527 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.658413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.30.160.62:46401)
0808 02:23:00.658460 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.658463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.658743 (+ 280us) server_negotiation.cc:407] Connection header received
0808 02:23:00.658844 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.658845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.658857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.658886 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:00.659570 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.660319 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.661381 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.661658 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.661725 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.664483 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:00.664484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:00.664484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:00.664492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:00.666230 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.666643 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.666645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.666645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.666705 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:00.666928 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:00.666929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:00.666929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:00.667057 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:00.667127 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.667357 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.667433 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:23:00.669586 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:23:00.669770 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:23:00.671759 6944 tablet_service.cc:1467] Processing CreateTablet for tablet 8838e76661d5457b8e702df9095b00f3 (DEFAULT_TABLE table=test-table [id=a3abd9020bcc429cb32b20561752af2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.671962 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8838e76661d5457b8e702df9095b00f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.673416 6797 tablet_service.cc:1467] Processing CreateTablet for tablet 8838e76661d5457b8e702df9095b00f3 (DEFAULT_TABLE table=test-table [id=a3abd9020bcc429cb32b20561752af2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.673538 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8838e76661d5457b8e702df9095b00f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.674937 7188 tablet_bootstrap.cc:492] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Bootstrap starting.
I20260808 02:23:00.675261 7188 tablet_bootstrap.cc:654] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.675516 7186 tablet_bootstrap.cc:492] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Bootstrap starting.
I20260808 02:23:00.675993 7186 tablet_bootstrap.cc:654] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.677143 7188 tablet_bootstrap.cc:492] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: No bootstrap required, opened a new log
I20260808 02:23:00.677187 7188 ts_tablet_manager.cc:1397] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:23:00.677330 7188 raft_consensus.cc:348] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.677395 7188 raft_consensus.cc:374] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.677415 7188 raft_consensus.cc:729] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.677459 7188 consensus_queue.cc:260] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.677541 7188 ts_tablet_manager.cc:1428] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.678922 7186 tablet_bootstrap.cc:492] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: No bootstrap required, opened a new log
I20260808 02:23:00.678972 7186 ts_tablet_manager.cc:1397] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 02:23:00.679131 7186 raft_consensus.cc:348] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.679194 7186 raft_consensus.cc:374] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.679214 7186 raft_consensus.cc:729] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc966872d29b4e66897e4629b2e8e52c, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.679251 7186 consensus_queue.cc:260] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.679338 7186 ts_tablet_manager.cc:1428] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.679553 7207 raft_consensus.cc:482] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:23:00.679597 7207 raft_consensus.cc:504] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.679731 7207 leader_election.cc:288] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727), 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.679998 7089 tablet_service.cc:1467] Processing CreateTablet for tablet 8838e76661d5457b8e702df9095b00f3 (DEFAULT_TABLE table=test-table [id=a3abd9020bcc429cb32b20561752af2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.680135 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8838e76661d5457b8e702df9095b00f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 02:23:00.680236 6892 leader_election.cc:341] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717): Not found: Tablet not found: 8838e76661d5457b8e702df9095b00f3
I20260808 02:23:00.682214 7187 tablet_bootstrap.cc:492] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Bootstrap starting.
I20260808 02:23:00.682637 7187 tablet_bootstrap.cc:654] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.683122 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e76661d5457b8e702df9095b00f3" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c" is_pre_election: true
I20260808 02:23:00.683207 6817 raft_consensus.cc:2454] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 0.
I20260808 02:23:00.683348 6895 leader_election.cc:302] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, dc966872d29b4e66897e4629b2e8e52c; no voters: 6fa0d0ce718c43959fc4fdaaa752bfac
I20260808 02:23:00.683470 7207 raft_consensus.cc:2781] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:23:00.683504 7207 raft_consensus.cc:482] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:23:00.683531 7207 raft_consensus.cc:3037] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.684244 7207 raft_consensus.cc:504] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.684387 7207 leader_election.cc:288] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Requested vote from peers dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727), 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717)
I20260808 02:23:00.684580 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e76661d5457b8e702df9095b00f3" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac"
W20260808 02:23:00.684804 6892 leader_election.cc:341] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717): Illegal state: must be running to vote when last-logged opid is not known
I20260808 02:23:00.684952 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e76661d5457b8e702df9095b00f3" candidate_uuid: "10560615f878403f974efb4e7d866aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c"
I20260808 02:23:00.685021 6817 raft_consensus.cc:3037] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.685720 6817 raft_consensus.cc:2454] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10560615f878403f974efb4e7d866aa8 in term 1.
I20260808 02:23:00.685882 6895 leader_election.cc:302] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, dc966872d29b4e66897e4629b2e8e52c; no voters: 6fa0d0ce718c43959fc4fdaaa752bfac
I20260808 02:23:00.686136 7207 raft_consensus.cc:2781] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:00.686184 7207 raft_consensus.cc:686] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Running, Role: LEADER
I20260808 02:23:00.686228 7207 consensus_queue.cc:237] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.686697 6656 catalog_manager.cc:5495] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10560615f878403f974efb4e7d866aa8 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "10560615f878403f974efb4e7d866aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } health_report { overall_health: UNKNOWN } } }
I20260808 02:23:00.694595 7187 tablet_bootstrap.cc:492] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: No bootstrap required, opened a new log
I20260808 02:23:00.694686 7187 ts_tablet_manager.cc:1397] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260808 02:23:00.694871 7187 raft_consensus.cc:348] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.694959 7187 raft_consensus.cc:374] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.694989 7187 raft_consensus.cc:729] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0d0ce718c43959fc4fdaaa752bfac, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.695039 7187 consensus_queue.cc:260] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.695158 7187 ts_tablet_manager.cc:1428] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.699512 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.696273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.30.160.2:33211)
0808 02:23:00.696337 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.696339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.696590 (+ 251us) server_negotiation.cc:407] Connection header received
0808 02:23:00.696670 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.696671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.696682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.696754 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.696913 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.697637 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.698721 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.699023 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.699091 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.699262 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.699296 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.699354 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.699417 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.699436 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:23:00.700202 6817 raft_consensus.cc:1261] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Refusing update from remote peer 10560615f878403f974efb4e7d866aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.700446 7110 raft_consensus.cc:3037] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.701265 7110 raft_consensus.cc:1261] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Refusing update from remote peer 10560615f878403f974efb4e7d866aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.701494 7207 consensus_queue.cc:1035] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.701907 7207 consensus_queue.cc:1035] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.705297 6657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.705406 6657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.706024 6657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Sending DeleteTablet for 3 replicas of tablet 8838e76661d5457b8e702df9095b00f3
I20260808 02:23:00.706431 6797 tablet_service.cc:1514] Processing DeleteTablet for tablet 8838e76661d5457b8e702df9095b00f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:54008
I20260808 02:23:00.706534 7089 tablet_service.cc:1514] Processing DeleteTablet for tablet 8838e76661d5457b8e702df9095b00f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42354
I20260808 02:23:00.706616 7212 tablet_replica.cc:330] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: stopping tablet replica
I20260808 02:23:00.706664 7212 raft_consensus.cc:2229] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.706694 7212 pending_rounds.cc:62] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Trying to abort 2 pending ops.
I20260808 02:23:00.706724 7212 pending_rounds.cc:68] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316094077746294784 op_type: WRITE_OP write_request { tablet_id: "8838e76661d5457b8e702df9095b00f3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\260\250\332\323\006\"8\n\ntest-admin\022*\n a3abd9020bcc429cb32b20561752af2f\020\001\030\001 \001(\001" signature: "<redacted>""\204O}\2010d\224\351\251\226\366\237\257u\334g\026\265Y2Lh\027\256\213gK\377+\220\013\255\2335Y\264\021\364C5\277\213\322d\320\314\241\370\003h\206\265\000g\220\017y\004\242_\311\361B\304" signing_key_seq_num: 0 } } request_id { client_id: "1758fe871858462d81ce209de9242fc7" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260808 02:23:00.706908 7212 pending_rounds.cc:68] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316094077691084800 op_type: NO_OP noop_request { }
I20260808 02:23:00.706954 7212 raft_consensus.cc:2866] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 02:23:00.706990 7212 raft_consensus.cc:2258] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.707063 7211 tablet_replica.cc:330] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: stopping tablet replica
I20260808 02:23:00.707119 7211 raft_consensus.cc:2229] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.707177 7211 raft_consensus.cc:2258] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.711064 6944 tablet_service.cc:1514] Processing DeleteTablet for tablet 8838e76661d5457b8e702df9095b00f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:40484
I20260808 02:23:00.711165 7210 tablet_replica.cc:330] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: stopping tablet replica
I20260808 02:23:00.711220 7210 raft_consensus.cc:2229] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:23:00.711277 7210 raft_consensus.cc:2258] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.711527 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:23:00.711633 7210 ts_tablet_manager.cc:1905] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:23:00.711686 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260808 02:23:00.712145 7264 consensus_queue.cc:1159] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "dc966872d29b4e66897e4629b2e8e52c" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260808 02:23:00.712450 7212 ts_tablet_manager.cc:1905] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.713600 7210 ts_tablet_manager.cc:1918] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:00.713663 7210 log.cc:1196] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/8838e76661d5457b8e702df9095b00f3
I20260808 02:23:00.713864 7212 ts_tablet_manager.cc:1918] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:00.713908 7210 ts_tablet_manager.cc:1939] T 8838e76661d5457b8e702df9095b00f3 P 10560615f878403f974efb4e7d866aa8: Deleting consensus metadata
I20260808 02:23:00.713927 7212 log.cc:1196] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/8838e76661d5457b8e702df9095b00f3
I20260808 02:23:00.714169 7212 ts_tablet_manager.cc:1939] T 8838e76661d5457b8e702df9095b00f3 P dc966872d29b4e66897e4629b2e8e52c: Deleting consensus metadata
I20260808 02:23:00.714524 6642 catalog_manager.cc:4841] TS 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211): tablet 8838e76661d5457b8e702df9095b00f3 (table test-table [id=a3abd9020bcc429cb32b20561752af2f]) successfully deleted
I20260808 02:23:00.714685 6643 catalog_manager.cc:4841] TS dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727): tablet 8838e76661d5457b8e702df9095b00f3 (table test-table [id=a3abd9020bcc429cb32b20561752af2f]) successfully deleted
I20260808 02:23:00.714884 7089 tablet_service.cc:1467] Processing CreateTablet for tablet 50cff6daf20f40bba4ba8bb0319e1eaa (DEFAULT_TABLE table=test-table [id=a361bebd39ee403e895301ef8eda4e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.714967 6944 tablet_service.cc:1467] Processing CreateTablet for tablet 50cff6daf20f40bba4ba8bb0319e1eaa (DEFAULT_TABLE table=test-table [id=a361bebd39ee403e895301ef8eda4e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.715003 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cff6daf20f40bba4ba8bb0319e1eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.715082 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cff6daf20f40bba4ba8bb0319e1eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.716281 7188 tablet_bootstrap.cc:492] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Bootstrap starting.
I20260808 02:23:00.716360 7187 tablet_bootstrap.cc:492] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Bootstrap starting.
I20260808 02:23:00.716624 7188 tablet_bootstrap.cc:654] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.716993 7187 tablet_bootstrap.cc:654] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.717124 6797 tablet_service.cc:1467] Processing CreateTablet for tablet 50cff6daf20f40bba4ba8bb0319e1eaa (DEFAULT_TABLE table=test-table [id=a361bebd39ee403e895301ef8eda4e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:00.717239 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cff6daf20f40bba4ba8bb0319e1eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:00.717707 7211 ts_tablet_manager.cc:1905] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.718415 7186 tablet_bootstrap.cc:492] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c: Bootstrap starting.
I20260808 02:23:00.718817 7186 tablet_bootstrap.cc:654] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:00.718920 7211 ts_tablet_manager.cc:1918] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:00.718982 7211 log.cc:1196] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/8838e76661d5457b8e702df9095b00f3
I20260808 02:23:00.719189 7211 ts_tablet_manager.cc:1939] T 8838e76661d5457b8e702df9095b00f3 P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting consensus metadata
I20260808 02:23:00.719488 6640 catalog_manager.cc:4841] TS 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717): tablet 8838e76661d5457b8e702df9095b00f3 (table test-table [id=a3abd9020bcc429cb32b20561752af2f]) successfully deleted
I20260808 02:23:00.720196 7187 tablet_bootstrap.cc:492] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: No bootstrap required, opened a new log
I20260808 02:23:00.720255 7187 ts_tablet_manager.cc:1397] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 02:23:00.720408 7187 raft_consensus.cc:348] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.720484 7187 raft_consensus.cc:374] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.720511 7187 raft_consensus.cc:729] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0d0ce718c43959fc4fdaaa752bfac, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.720554 7187 consensus_queue.cc:260] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.720651 7187 ts_tablet_manager.cc:1428] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.721736 7186 tablet_bootstrap.cc:492] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c: No bootstrap required, opened a new log
I20260808 02:23:00.721791 7186 ts_tablet_manager.cc:1397] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 02:23:00.721915 7186 raft_consensus.cc:348] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.721979 7186 raft_consensus.cc:374] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.722007 7186 raft_consensus.cc:729] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc966872d29b4e66897e4629b2e8e52c, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.722046 7186 consensus_queue.cc:260] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.722144 7186 ts_tablet_manager.cc:1428] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.725024 7188 tablet_bootstrap.cc:492] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: No bootstrap required, opened a new log
I20260808 02:23:00.725086 7188 ts_tablet_manager.cc:1397] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.000s
I20260808 02:23:00.725201 7188 raft_consensus.cc:348] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.725278 7188 raft_consensus.cc:374] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:00.725308 7188 raft_consensus.cc:729] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10560615f878403f974efb4e7d866aa8, State: Initialized, Role: FOLLOWER
I20260808 02:23:00.725368 7188 consensus_queue.cc:260] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.725471 7188 ts_tablet_manager.cc:1428] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:00.855418 7195 raft_consensus.cc:482] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:23:00.855553 7195 raft_consensus.cc:504] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.855947 7195 leader_election.cc:288] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727), 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211)
I20260808 02:23:00.860047 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.856218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:34331 (local address 127.30.160.2:33211)
0808 02:23:00.856268 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.856270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.856361 (+ 91us) server_negotiation.cc:407] Connection header received
0808 02:23:00.856551 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.856553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.856563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.856618 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.856782 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.857557 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.859113 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.859691 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.859753 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.859764 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.859934 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.859975 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:23:00.860047 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.856229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:60355 (local address 127.30.160.1:42727)
0808 02:23:00.856279 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.856281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.856361 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:23:00.856421 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.856422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.856432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.856510 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:00.856667 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.857464 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.859114 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.859668 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.859748 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.859757 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.859936 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.859975 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:23:00.860114 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.856135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42727 (local address 127.30.160.3:60355)
0808 02:23:00.856268 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.856280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.856354 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.856522 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.856523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.856525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.856578 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.856579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.857548 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.857549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.859023 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.859026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.859748 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.859749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.859829 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.859837 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.859918 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.859979 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":81,"thread_start_us":51,"threads_started":1}
I20260808 02:23:00.860113 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.856135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33211 (local address 127.30.160.3:34331)
0808 02:23:00.856268 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.856280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.856354 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.856642 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.856643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.856645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:00.856711 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.856713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.857622 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.857623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.859023 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.859026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.859751 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.859752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.859829 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.859838 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.859918 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.859979 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":66,"thread_start_us":67,"threads_started":1}
I20260808 02:23:00.860424 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cff6daf20f40bba4ba8bb0319e1eaa" candidate_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c" is_pre_election: true
I20260808 02:23:00.860435 6964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cff6daf20f40bba4ba8bb0319e1eaa" candidate_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10560615f878403f974efb4e7d866aa8" is_pre_election: true
I20260808 02:23:00.860522 6964 raft_consensus.cc:2454] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa0d0ce718c43959fc4fdaaa752bfac in term 0.
I20260808 02:23:00.860539 6817 raft_consensus.cc:2454] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa0d0ce718c43959fc4fdaaa752bfac in term 0.
I20260808 02:23:00.860689 7040 leader_election.cc:302] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10560615f878403f974efb4e7d866aa8, 6fa0d0ce718c43959fc4fdaaa752bfac; no voters:
I20260808 02:23:00.860854 7195 raft_consensus.cc:2781] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:23:00.860910 7195 raft_consensus.cc:482] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:23:00.860934 7195 raft_consensus.cc:3037] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.861735 7195 raft_consensus.cc:504] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.861899 7195 leader_election.cc:288] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [CANDIDATE]: Term 1 election: Requested vote from peers dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727), 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211)
I20260808 02:23:00.862046 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cff6daf20f40bba4ba8bb0319e1eaa" candidate_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc966872d29b4e66897e4629b2e8e52c"
I20260808 02:23:00.862044 6964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cff6daf20f40bba4ba8bb0319e1eaa" candidate_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10560615f878403f974efb4e7d866aa8"
I20260808 02:23:00.862126 6817 raft_consensus.cc:3037] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.862144 6964 raft_consensus.cc:3037] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:00.862757 6817 raft_consensus.cc:2454] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa0d0ce718c43959fc4fdaaa752bfac in term 1.
I20260808 02:23:00.862788 6964 raft_consensus.cc:2454] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa0d0ce718c43959fc4fdaaa752bfac in term 1.
I20260808 02:23:00.862962 7041 leader_election.cc:302] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa0d0ce718c43959fc4fdaaa752bfac, dc966872d29b4e66897e4629b2e8e52c; no voters:
I20260808 02:23:00.863073 7195 raft_consensus.cc:2781] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:00.863204 7195 raft_consensus.cc:686] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 LEADER]: Becoming Leader. State: Replica: 6fa0d0ce718c43959fc4fdaaa752bfac, State: Running, Role: LEADER
I20260808 02:23:00.863302 7195 consensus_queue.cc:237] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } }
I20260808 02:23:00.864038 6657 catalog_manager.cc:5495] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3). New cstate: current_term: 1 leader_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa0d0ce718c43959fc4fdaaa752bfac" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 39717 } health_report { overall_health: HEALTHY } } }
I20260808 02:23:00.871309 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.868774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.30.160.3:39717)
0808 02:23:00.868985 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.868986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.868993 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:23:00.869026 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.869026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.869043 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.869111 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.869270 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.869796 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.870529 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.870732 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.870785 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.870976 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.871079 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.871134 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.871175 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.871192 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
I20260808 02:23:00.876930 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.874007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:34857 (local address 127.30.160.2:33211)
0808 02:23:00.874059 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.874062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.874095 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:23:00.874182 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.874183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.874198 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.874305 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.874466 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.875006 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.876087 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.876312 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.876406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.876626 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.876683 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.876754 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.876824 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.876847 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:23:00.876951 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.873830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:33211 (local address 127.30.160.3:34857)
0808 02:23:00.874029 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.874044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.874113 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.874310 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.874311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.874313 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:23:00.874378 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.874380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.875122 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.875123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.875972 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.875981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.876408 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.876409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.876499 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.876514 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.876757 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.876760 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.876819 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.876866 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":125,"threads_started":1}
I20260808 02:23:00.879390 6817 raft_consensus.cc:1261] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Refusing update from remote peer 6fa0d0ce718c43959fc4fdaaa752bfac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.879405 6964 raft_consensus.cc:1261] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Refusing update from remote peer 6fa0d0ce718c43959fc4fdaaa752bfac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:00.879719 7271 consensus_queue.cc:1035] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc966872d29b4e66897e4629b2e8e52c" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.879894 7195 consensus_queue.cc:1035] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [LEADER]: Connected to new peer: Peer: permanent_uuid: "10560615f878403f974efb4e7d866aa8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:00.887483 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.885024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:58669 (local address 127.30.160.3:39717)
0808 02:23:00.885091 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:00.885094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:00.885104 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:00.885127 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:00.885128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:00.885141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:00.885201 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:00.885321 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.885896 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.886693 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.886925 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.887016 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.887211 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.887244 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.887298 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:00.887394 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 02:23:00.887416 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:23:00.887552 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.884576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:39717 (local address 127.30.160.2:58669)
0808 02:23:00.884633 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.884643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:00.884682 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:00.885205 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:00.885207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:00.885208 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:23:00.885253 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.885254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.885985 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.885987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.886590 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:00.886592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:00.887045 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:00.887047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:00.887114 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:00.887122 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:23:00.887322 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:23:00.887325 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:00.887389 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 02:23:00.887451 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 02:23:00.894462 6656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.894582 6656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:00.895283 6656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daa45bf98af432a8f1e771d88008f3f: Sending DeleteTablet for 3 replicas of tablet 50cff6daf20f40bba4ba8bb0319e1eaa
I20260808 02:23:00.895589 6797 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cff6daf20f40bba4ba8bb0319e1eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:54008
I20260808 02:23:00.895629 7089 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cff6daf20f40bba4ba8bb0319e1eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42354
I20260808 02:23:00.895638 6944 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cff6daf20f40bba4ba8bb0319e1eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:00 UTC) from {username='oryx', principal='oryx/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:40484
I20260808 02:23:00.895728 7211 tablet_replica.cc:330] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: stopping tablet replica
I20260808 02:23:00.895733 7212 tablet_replica.cc:330] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c: stopping tablet replica
I20260808 02:23:00.895792 7212 raft_consensus.cc:2229] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.895792 7211 raft_consensus.cc:2229] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:23:00.895828 7212 raft_consensus.cc:2258] T 50cff6daf20f40bba4ba8bb0319e1eaa P dc966872d29b4e66897e4629b2e8e52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.895875 7211 raft_consensus.cc:2258] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.895921 7210 tablet_replica.cc:330] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: stopping tablet replica
I20260808 02:23:00.895977 7210 raft_consensus.cc:2229] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:00.896010 7210 raft_consensus.cc:2258] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:00.896484 7211 ts_tablet_manager.cc:1905] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.896893 7210 ts_tablet_manager.cc:1905] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:00.896900 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 6731
I20260808 02:23:00.898277 7211 ts_tablet_manager.cc:1918] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:23:00.898337 7211 log.cc:1196] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/50cff6daf20f40bba4ba8bb0319e1eaa
I20260808 02:23:00.898594 7211 ts_tablet_manager.cc:1939] T 50cff6daf20f40bba4ba8bb0319e1eaa P 6fa0d0ce718c43959fc4fdaaa752bfac: Deleting consensus metadata
I20260808 02:23:00.898572 7210 ts_tablet_manager.cc:1918] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:23:00.898639 7210 log.cc:1196] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/50cff6daf20f40bba4ba8bb0319e1eaa
I20260808 02:23:00.898865 7210 ts_tablet_manager.cc:1939] T 50cff6daf20f40bba4ba8bb0319e1eaa P 10560615f878403f974efb4e7d866aa8: Deleting consensus metadata
I20260808 02:23:00.899930 6640 catalog_manager.cc:4841] TS 6fa0d0ce718c43959fc4fdaaa752bfac (127.30.160.3:39717): tablet 50cff6daf20f40bba4ba8bb0319e1eaa (table test-table [id=a361bebd39ee403e895301ef8eda4e8e]) successfully deleted
I20260808 02:23:00.900461 6642 catalog_manager.cc:4841] TS 10560615f878403f974efb4e7d866aa8 (127.30.160.2:33211): tablet 50cff6daf20f40bba4ba8bb0319e1eaa (table test-table [id=a361bebd39ee403e895301ef8eda4e8e]) successfully deleted
W20260808 02:23:00.907689 6643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:42727 (error 108)
I20260808 02:23:00.907982 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 6876
I20260808 02:23:00.914876 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:00.908077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42727 (local address 127.0.0.1:45976)
0808 02:23:00.914769 (+ 6692us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:00.914831 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6674,"thread_start_us":45,"threads_started":1}
W20260808 02:23:00.914987 6643 catalog_manager.cc:4584] TS dc966872d29b4e66897e4629b2e8e52c (127.30.160.1:42727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50cff6daf20f40bba4ba8bb0319e1eaa: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42727: connect: Connection refused (error 111)
I20260808 02:23:00.915181 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7022
I20260808 02:23:00.921066 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 6624
2026-08-08T02:23:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2674 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[7285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[7285](info): set up 2 sockets
Aug 08 02:23:00 dist-test-slave-zkpd krb5kdc[7285](info): commencing operation
krb5kdc: starting...
W20260808 02:23:02.000255 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:23:02 dist-test-slave-zkpd krb5kdc[7285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:23:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:23:02.120458 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:35815
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:02.204231 7298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:02.204456 7298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:02.204490 7298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:02.205607 7298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:23:02.205673 7298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:02.205698 7298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:02.205734 7298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:23:02.205765 7298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:23:02.207252 7298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:02.207533 7298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:02.207762 7298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:02.212589 7303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:02.212610 7304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:02.212634 7307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:02.212877 7298 server_base.cc:1034] running on GCE node
I20260808 02:23:02.213104 7298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:02.213433 7298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:02.214599 7298 hybrid_clock.cc:648] HybridClock initialized: now 1786155782214588 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T02:23:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1295 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:23:02 dist-test-slave-zkpd krb5kdc[7312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:23:02 dist-test-slave-zkpd krb5kdc[7312](info): set up 2 sockets
Aug 08 02:23:02 dist-test-slave-zkpd krb5kdc[7312](info): commencing operation
krb5kdc: starting...
W20260808 02:23:03.284107 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:23:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:23:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:23:03.402760 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45279
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:03.487253 7325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:03.487449 7325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:03.487471 7325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:03.488623 7325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:23:03.488665 7325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:03.488680 7325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:03.488693 7325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:23:03.488713 7325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:23:03.490101 7325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:03.490283 7325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:03.490442 7325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:03.495208 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:03.495267 7325 server_base.cc:1034] running on GCE node
W20260808 02:23:03.495221 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:03.495237 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:03.495694 7325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:03.496047 7325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:03.497210 7325 hybrid_clock.cc:648] HybridClock initialized: now 1786155783497205 us; error 38 us; skew 500 ppm
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:03.504285 7325 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:23:03.504666 7325 webserver.cc:460] Webserver started at http://127.30.160.62:35763/ using document root <none> and password file <none>
I20260808 02:23:03.504851 7325 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:03.504891 7325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:03.504977 7325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:03.505961 7325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "591c5c334fcf4b65ad43aa26b5ff88c1"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "1ac699bec28e1d64d07a33dd481f179a"
server_key_iv: "8707c74fcfb647cf037fc76bccb65cde"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.506256 7325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "591c5c334fcf4b65ad43aa26b5ff88c1"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "1ac699bec28e1d64d07a33dd481f179a"
server_key_iv: "8707c74fcfb647cf037fc76bccb65cde"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.507524 7325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:23:03.508247 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.508448 7325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.508539 7325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "591c5c334fcf4b65ad43aa26b5ff88c1"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "1ac699bec28e1d64d07a33dd481f179a"
server_key_iv: "8707c74fcfb647cf037fc76bccb65cde"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.508594 7325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:03.530719 7325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:03.533041 7325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:03.533218 7325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:03.537173 7325 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:45279
I20260808 02:23:03.537246 7393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:45279 every 8 connection(s)
I20260808 02:23:03.537600 7325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:23:03.538514 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:03.540884 7394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Bootstrap starting.
I20260808 02:23:03.541464 7394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:03.541759 7394 log.cc:824] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:03.542483 7394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: No bootstrap required, opened a new log
I20260808 02:23:03.544124 7394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:03.544281 7394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:03.544351 7394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c5c334fcf4b65ad43aa26b5ff88c1, State: Initialized, Role: FOLLOWER
I20260808 02:23:03.544530 7394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:03.544616 7394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:23:03.544661 7394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:23:03.544710 7394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:03.545261 7394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:03.545387 7394 leader_election.cc:302] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591c5c334fcf4b65ad43aa26b5ff88c1; no voters:
I20260808 02:23:03.545562 7394 leader_election.cc:288] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:23:03.545612 7398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:03.545809 7394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:23:03.546027 7398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 LEADER]: Becoming Leader. State: Replica: 591c5c334fcf4b65ad43aa26b5ff88c1, State: Running, Role: LEADER
I20260808 02:23:03.546195 7398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:03.546602 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } } }
I20260808 02:23:03.546630 7400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591c5c334fcf4b65ad43aa26b5ff88c1. Latest consensus state: current_term: 1 leader_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } } }
I20260808 02:23:03.546680 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:03.546706 7400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:03.546895 7408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:23:03.547469 7408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:23:03.547695 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7325
I20260808 02:23:03.547824 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:23:03.548049 31360 external_mini_cluster.cc:1428] Setting key 30ecb394e8a4374efa5019f762353db0
I20260808 02:23:03.549779 7408 catalog_manager.cc:1347] Generated new cluster ID: e6504621d1a24090ab4faf1cacefd531
I20260808 02:23:03.549842 7408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:03.559954 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:03.548701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.30.160.62:45279)
0808 02:23:03.548876 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:03.548885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:03.548915 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:23:03.549026 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:03.549033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:03.549090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:03.549270 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:03.550013 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.551444 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.552522 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.553166 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.553387 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:03.556261 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:03.556265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:03.556265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:03.556277 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:03.558204 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:03.558618 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:03.558621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:03.558622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:03.558723 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:03.558949 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:03.558950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:03.558950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:03.559222 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:03.559367 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:03.559483 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 02:23:03.559702 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260808 02:23:03.566670 7408 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:23:03.567354 7408 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:23:03.576309 7408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Generated new TSK 0
I20260808 02:23:03.576484 7408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:23:03.605075 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45279
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:03.686867 7422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:03.687074 7422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:03.687098 7422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:03.688203 7422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:03.688246 7422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:03.688290 7422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:23:03.689757 7422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:03.689932 7422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:03.690102 7422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:03.695065 7427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:03.695081 7430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:03.695078 7428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:03.695361 7422 server_base.cc:1034] running on GCE node
I20260808 02:23:03.695570 7422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:03.695823 7422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:03.696987 7422 hybrid_clock.cc:648] HybridClock initialized: now 1786155783696972 us; error 26 us; skew 500 ppm
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:03.699401 7422 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:23:03.699837 7422 webserver.cc:460] Webserver started at http://127.30.160.1:35055/ using document root <none> and password file <none>
I20260808 02:23:03.700102 7422 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:03.700173 7422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:03.700325 7422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:03.701340 7422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "16624dad6e624ee39562c3a41198f479"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "980f0378d284bb7a07b47a80ee28ee84"
server_key_iv: "af6ecb892130f2009a35b240ad1f6b5e"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.701699 7422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "16624dad6e624ee39562c3a41198f479"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "980f0378d284bb7a07b47a80ee28ee84"
server_key_iv: "af6ecb892130f2009a35b240ad1f6b5e"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.703116 7422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:23:03.703773 7437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.703975 7422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:03.704072 7422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "16624dad6e624ee39562c3a41198f479"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "980f0378d284bb7a07b47a80ee28ee84"
server_key_iv: "af6ecb892130f2009a35b240ad1f6b5e"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.704175 7422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:03.726917 7422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:03.728776 7422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:03.728950 7422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:03.729184 7422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:03.729503 7422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:03.729557 7422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.729598 7422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:03.729627 7422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.735378 7422 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:36315
I20260808 02:23:03.735430 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:36315 every 8 connection(s)
I20260808 02:23:03.735728 7422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:23:03.739838 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7422
I20260808 02:23:03.739974 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:23:03.740198 31360 external_mini_cluster.cc:1428] Setting key b2252952f8ae91502d9e50aac402c4ae
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:03.745734 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:03.736465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:53177 (local address 127.30.160.62:45279)
0808 02:23:03.736525 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:03.736527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:03.736734 (+ 207us) server_negotiation.cc:407] Connection header received
0808 02:23:03.737272 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:03.737274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:03.737291 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:03.737359 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:03.738373 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.738935 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.740146 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.740412 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.740475 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:03.742815 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:03.742817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:03.742817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:03.742829 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:03.744335 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:03.744757 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:03.744758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:03.744758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:03.744820 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:03.745084 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:03.745085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:03.745086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:03.745202 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:03.745278 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:03.745550 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 02:23:03.745644 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:23:03.745970 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:03.736429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:53177)
0808 02:23:03.736605 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:03.736649 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:03.737164 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:03.737427 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:03.737431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:03.738076 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:03.738266 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:03.738271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.739036 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.739037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:03.740055 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:03.740060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.740503 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.740504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:03.740675 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:03.741173 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:03.741186 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:03.742720 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:03.744435 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:03.744443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:03.744445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:03.744669 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:03.744885 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:03.744886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:03.744887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:03.745025 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:03.745270 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:03.745277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:03.745417 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 02:23:03.745527 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:23:03.745678 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260808 02:23:03.746537 7559 heartbeater.cc:347] Connected to a master server at 127.30.160.62:45279
I20260808 02:23:03.746641 7559 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:03.746860 7559 heartbeater.cc:511] Master 127.30.160.62:45279 requested a full tablet report, sending...
I20260808 02:23:03.747421 7357 ts_manager.cc:194] Registered new tserver with Master: 16624dad6e624ee39562c3a41198f479 (127.30.160.1:36315)
I20260808 02:23:03.748402 7357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:53177
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:23:03.764808 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45279
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:03.846194 7564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:03.846386 7564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:03.846410 7564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:03.847610 7564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:03.847651 7564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:03.847677 7564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:23:03.849102 7564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:03.849280 7564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:03.849447 7564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:03.855659 7572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:03.855655 7570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:03.855978 7564 server_base.cc:1034] running on GCE node
W20260808 02:23:03.859380 7569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:03.859655 7564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:03.859951 7564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:03.861137 7564 hybrid_clock.cc:648] HybridClock initialized: now 1786155783861125 us; error 69 us; skew 500 ppm
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:03.864193 7564 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:23:03.864668 7564 webserver.cc:460] Webserver started at http://127.30.160.2:40031/ using document root <none> and password file <none>
I20260808 02:23:03.865036 7564 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:03.865164 7564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:03.865355 7564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:03.867136 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "d7bd34c34afc455c900351bab6e1a8dc"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "c2c40cbc05e1569f67b0835e50b8f8f0"
server_key_iv: "d9e1982cbbc4cc5b6334254128d15541"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.867586 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "d7bd34c34afc455c900351bab6e1a8dc"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "c2c40cbc05e1569f67b0835e50b8f8f0"
server_key_iv: "d9e1982cbbc4cc5b6334254128d15541"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.869498 7564 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:23:03.870529 7579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.870741 7564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:03.870844 7564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "d7bd34c34afc455c900351bab6e1a8dc"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "c2c40cbc05e1569f67b0835e50b8f8f0"
server_key_iv: "d9e1982cbbc4cc5b6334254128d15541"
server_key_version: "encryptionkey@0"
I20260808 02:23:03.870939 7564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:03.904695 7564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:03.906665 7564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:03.906868 7564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:03.907119 7564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:03.907445 7564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:03.907501 7564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.907544 7564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:03.907570 7564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:03.913817 7564 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38063
I20260808 02:23:03.913874 7695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38063 every 8 connection(s)
I20260808 02:23:03.914175 7564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:23:03.919627 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7564
I20260808 02:23:03.919745 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:23:03.919978 31360 external_mini_cluster.cc:1428] Setting key e8ee26962fcb7cb54d9aa9747a92d2da
Aug 08 02:23:03 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:03.923512 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:03.914833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:48357 (local address 127.30.160.62:45279)
0808 02:23:03.914904 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:03.914906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:03.915191 (+ 285us) server_negotiation.cc:407] Connection header received
0808 02:23:03.915654 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:03.915655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:03.915665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:03.915720 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:03.916710 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.917187 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.918135 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.918320 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.918370 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:03.920666 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:03.920668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:03.920668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:03.920677 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:03.921970 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:03.922520 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:03.922521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:03.922521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:03.922586 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:03.922860 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:03.922861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:03.922862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:03.922982 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:03.923061 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:03.923336 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 02:23:03.923422 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:23:03.923712 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:03.914838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:48357)
0808 02:23:03.915057 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:03.915097 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:03.915570 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:03.915786 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:03.915789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:03.916454 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:03.916634 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:03.916639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.917266 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.917266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:03.918062 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:03.918064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:03.918393 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:03.918394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:03.918489 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:03.918924 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:03.918932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:03.920568 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:03.922048 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:03.922056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:03.922058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:03.922417 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:03.922657 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:03.922658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:03.922658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:03.922779 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:03.923052 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:03.923059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:03.923198 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 02:23:03.923302 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:23:03.923466 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260808 02:23:03.924188 7696 heartbeater.cc:347] Connected to a master server at 127.30.160.62:45279
I20260808 02:23:03.924296 7696 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:03.924479 7696 heartbeater.cc:511] Master 127.30.160.62:45279 requested a full tablet report, sending...
I20260808 02:23:03.924898 7357 ts_manager.cc:194] Registered new tserver with Master: d7bd34c34afc455c900351bab6e1a8dc (127.30.160.2:38063)
I20260808 02:23:03.925274 7357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:48357
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:23:03.946887 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45279
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:04.029704 7701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:04.029904 7701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:04.029927 7701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:04.031011 7701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:04.031044 7701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:04.031069 7701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:23:04.032565 7701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:04.032742 7701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:04.032922 7701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:04.037729 7706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.037770 7709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.037990 7701 server_base.cc:1034] running on GCE node
W20260808 02:23:04.037748 7707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.038275 7701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:04.038569 7701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:04.039731 7701 hybrid_clock.cc:648] HybridClock initialized: now 1786155784039717 us; error 45 us; skew 500 ppm
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:04.042168 7701 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:23:04.042624 7701 webserver.cc:460] Webserver started at http://127.30.160.3:45935/ using document root <none> and password file <none>
I20260808 02:23:04.042865 7701 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:04.042935 7701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:04.043076 7701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:04.044199 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "f1ac904608f048c4a846e81a29cd2374"
format_stamp: "Formatted at 2026-08-08 02:23:04 on dist-test-slave-zkpd"
server_key: "56fdecc6e7c3437eb0459911285d9c69"
server_key_iv: "add083e5d6ff38626ece8cb28fa5b356"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.044566 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "f1ac904608f048c4a846e81a29cd2374"
format_stamp: "Formatted at 2026-08-08 02:23:04 on dist-test-slave-zkpd"
server_key: "56fdecc6e7c3437eb0459911285d9c69"
server_key_iv: "add083e5d6ff38626ece8cb28fa5b356"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.045986 7701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:04.046794 7716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.046977 7701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:04.047079 7701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "f1ac904608f048c4a846e81a29cd2374"
format_stamp: "Formatted at 2026-08-08 02:23:04 on dist-test-slave-zkpd"
server_key: "56fdecc6e7c3437eb0459911285d9c69"
server_key_iv: "add083e5d6ff38626ece8cb28fa5b356"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.047189 7701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:04.061978 7701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:04.064252 7701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:04.064364 7701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:04.064545 7701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:04.064781 7701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:04.064814 7701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.064836 7701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:04.064848 7701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.070581 7701 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:45375
I20260808 02:23:04.070657 7830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:45375 every 8 connection(s)
I20260808 02:23:04.070959 7701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:23:04.071880 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7701
I20260808 02:23:04.072029 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:23:04.072219 31360 external_mini_cluster.cc:1428] Setting key 7cd7c6eccde969549a6fb33b0277b643
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:04.080559 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.071662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:45835 (local address 127.30.160.62:45279)
0808 02:23:04.071744 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.071747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.072041 (+ 294us) server_negotiation.cc:407] Connection header received
0808 02:23:04.072696 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.072697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.072709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.072766 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.073786 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.074290 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.075294 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.075506 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.075580 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.077919 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:04.077920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:04.077921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:04.077928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:04.079148 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:04.079555 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:04.079556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:04.079556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:04.079624 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:04.079887 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:04.079888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:04.079888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:04.080027 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:04.080101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:04.080379 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 02:23:04.080464 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:23:04.080757 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.071638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:45835)
0808 02:23:04.071853 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.071927 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.072600 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.072839 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.072843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.073477 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.073665 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.073670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.074388 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.074389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.075185 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.075188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.075601 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.075603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.075737 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.076299 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.076326 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.077815 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:04.079246 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:04.079253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:04.079255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:04.079478 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:04.079694 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:04.079695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:04.079695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:04.079810 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:04.080106 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:04.080113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:04.080246 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 02:23:04.080355 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:23:04.080513 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260808 02:23:04.081233 7832 heartbeater.cc:347] Connected to a master server at 127.30.160.62:45279
I20260808 02:23:04.081377 7832 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:04.081568 7832 heartbeater.cc:511] Master 127.30.160.62:45279 requested a full tablet report, sending...
I20260808 02:23:04.081983 7357 ts_manager.cc:194] Registered new tserver with Master: f1ac904608f048c4a846e81a29cd2374 (127.30.160.3:45375)
I20260808 02:23:04.082348 7357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:45835
I20260808 02:23:04.084007 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab.
Entry for principal oryx/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab.
WARNING: no policy specified for oryx/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab.
Entry for principal oryx/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab.
WARNING: no policy specified for oryx/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab.
Entry for principal oryx/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab.
I20260808 02:23:04.156432 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7422
I20260808 02:23:04.162688 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7564
I20260808 02:23:04.168478 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7701
I20260808 02:23:04.174480 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7325
I20260808 02:23:04.180670 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45279
--webserver_interface=127.30.160.62
--webserver_port=35763
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:04.262023 7843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:04.262269 7843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:04.262308 7843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:04.263432 7843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:23:04.263509 7843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:04.263535 7843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:04.263577 7843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:23:04.263607 7843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:23:04.265127 7843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=35763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:04.265389 7843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:04.265614 7843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:04.270255 7849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.270254 7848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.270283 7851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.270848 7843 server_base.cc:1034] running on GCE node
I20260808 02:23:04.271090 7843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:04.271394 7843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:04.272550 7843 hybrid_clock.cc:648] HybridClock initialized: now 1786155784272540 us; error 37 us; skew 500 ppm
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:04.275048 7843 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:23:04.275523 7843 webserver.cc:460] Webserver started at http://127.30.160.62:35763/ using document root <none> and password file <none>
I20260808 02:23:04.275774 7843 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:04.275859 7843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:04.277211 7843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:04.277884 7858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.278054 7843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:04.278163 7843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "591c5c334fcf4b65ad43aa26b5ff88c1"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "1ac699bec28e1d64d07a33dd481f179a"
server_key_iv: "8707c74fcfb647cf037fc76bccb65cde"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.278496 7843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:04.289444 7843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:04.291168 7843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:04.291332 7843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:04.295463 7843 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:45279
I20260808 02:23:04.295495 7910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:45279 every 8 connection(s)
I20260808 02:23:04.295867 7843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:23:04.297353 7911 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:23:04.297924 7911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Bootstrap starting.
I20260808 02:23:04.305298 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7843
I20260808 02:23:04.305641 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:36315
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:45279
--webserver_port=35055
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:23:04.306514 7911 log.cc:824] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:04.308759 7911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:23:04.309041 7911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Bootstrap complete.
I20260808 02:23:04.311352 7911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:04.311594 7911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c5c334fcf4b65ad43aa26b5ff88c1, State: Initialized, Role: FOLLOWER
I20260808 02:23:04.311774 7911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:04.311853 7911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:23:04.311924 7911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:23:04.311978 7911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:23:04.312925 7911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:04.313048 7911 leader_election.cc:302] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591c5c334fcf4b65ad43aa26b5ff88c1; no voters:
I20260808 02:23:04.313241 7911 leader_election.cc:288] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:23:04.313277 7915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:23:04.313393 7915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [term 2 LEADER]: Becoming Leader. State: Replica: 591c5c334fcf4b65ad43aa26b5ff88c1, State: Running, Role: LEADER
I20260808 02:23:04.313524 7911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:23:04.313555 7915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } }
I20260808 02:23:04.313865 7915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591c5c334fcf4b65ad43aa26b5ff88c1. Latest consensus state: current_term: 2 leader_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } } }
I20260808 02:23:04.313868 7916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591c5c334fcf4b65ad43aa26b5ff88c1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45279 } } }
I20260808 02:23:04.313966 7915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:04.314003 7916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:04.314162 7921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:23:04.314675 7921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:23:04.314832 7921 catalog_manager.cc:1259] Loaded cluster ID: e6504621d1a24090ab4faf1cacefd531
I20260808 02:23:04.315034 7921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:23:04.316249 7921 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:23:04.316433 7921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 591c5c334fcf4b65ad43aa26b5ff88c1: Loaded TSK: 0
I20260808 02:23:04.316645 7921 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:23:04.397507 7914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:04.397727 7914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:04.397765 7914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:04.398921 7914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:04.398989 7914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:04.399035 7914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:23:04.400555 7914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.1.keytab
--principal=oryx/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:36315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=35055
--tserver_master_addrs=127.30.160.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:04.400826 7914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:04.401044 7914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:04.406791 7938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.406792 7937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.406781 7940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.407078 7914 server_base.cc:1034] running on GCE node
I20260808 02:23:04.407295 7914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:04.407538 7914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:04.408699 7914 hybrid_clock.cc:648] HybridClock initialized: now 1786155784408687 us; error 34 us; skew 500 ppm
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155784, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:04.411286 7914 init.cc:377] Logged in from keytab as oryx/127.30.160.1@KRBTEST.COM (short username oryx)
I20260808 02:23:04.411779 7914 webserver.cc:460] Webserver started at http://127.30.160.1:35055/ using document root <none> and password file <none>
I20260808 02:23:04.412066 7914 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:04.412150 7914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:04.413569 7914 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:04.414268 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.414461 7914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:04.414547 7914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "16624dad6e624ee39562c3a41198f479"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "980f0378d284bb7a07b47a80ee28ee84"
server_key_iv: "af6ecb892130f2009a35b240ad1f6b5e"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.414886 7914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:04.422017 7914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:04.424122 7914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:04.424278 7914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:04.424486 7914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:04.424775 7914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:04.424837 7914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.424881 7914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:04.424906 7914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.430451 7914 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:36315
I20260808 02:23:04.430564 8063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:36315 every 8 connection(s)
I20260808 02:23:04.430884 7914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:23:04.432191 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 7914
I20260808 02:23:04.432504 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:38063
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:45279
--webserver_port=40031
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.440924 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.431588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:54901)
0808 02:23:04.431794 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.431846 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.432488 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.433031 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.433035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.433862 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.434137 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.434151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.435359 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.435360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.436237 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.436240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.437165 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.437167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.437305 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.437907 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.437918 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.440726 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
W20260808 02:23:04.441128 8064 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:45279 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.160.62:45279: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
W20260808 02:23:04.441476 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.431723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:54901 (local address 127.30.160.62:45279)
0808 02:23:04.431933 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.431941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.431976 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:23:04.432750 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.432756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.432824 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.432961 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.434302 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.435230 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.436347 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.436964 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.437072 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.440912 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:54901: BlockingRecv error: recv got EOF from 127.30.160.1:54901 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.447031 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.441336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:45095)
0808 02:23:04.441621 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.441632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.441673 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.441892 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.441892 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.442037 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.442088 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.442089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.443047 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.443048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.443863 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.443867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.444413 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.444414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.444483 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.444967 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.444972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.446961 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":67,"threads_started":1}
W20260808 02:23:04.447118 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.441500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:45095 (local address 127.30.160.62:45279)
0808 02:23:04.441740 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.441742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.441752 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:04.441775 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.441776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.441790 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.441874 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.442173 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.442951 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.443993 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.444328 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.444407 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.447059 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:45095: BlockingRecv error: recv got EOF from 127.30.160.1:45095 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":63,"threads_started":1}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.452785 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.447680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:34715)
0808 02:23:04.447778 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.447788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.447839 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.447996 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.447997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.448137 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.448167 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.448168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.449067 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.449068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.449876 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.449878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.450326 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.450328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.450381 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.450802 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.450807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.452719 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260808 02:23:04.452874 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.447739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:34715 (local address 127.30.160.62:45279)
0808 02:23:04.447777 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.447779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.447827 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:23:04.447915 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.447916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.447928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.447980 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.448250 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.448959 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.449966 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.450247 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.450318 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.452820 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:34715: BlockingRecv error: recv got EOF from 127.30.160.1:34715 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 02:23:04.453107 8064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:23:04.520531 8068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:04.520744 8068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:04.520779 8068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:04.521879 8068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:04.521946 8068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:04.521992 8068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:23:04.523447 8068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.2.keytab
--principal=oryx/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:38063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=40031
--tserver_master_addrs=127.30.160.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:04.523710 8068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:04.523962 8068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:04.528904 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.528919 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.529054 8068 server_base.cc:1034] running on GCE node
W20260808 02:23:04.528923 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.529332 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:04.529601 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:04.530761 8068 hybrid_clock.cc:648] HybridClock initialized: now 1786155784530741 us; error 35 us; skew 500 ppm
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155784, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:04.533155 8068 init.cc:377] Logged in from keytab as oryx/127.30.160.2@KRBTEST.COM (short username oryx)
I20260808 02:23:04.533573 8068 webserver.cc:460] Webserver started at http://127.30.160.2:40031/ using document root <none> and password file <none>
I20260808 02:23:04.533810 8068 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:04.533895 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:04.535161 8068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:23:04.535698 8086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.535863 8068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:23:04.535989 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "d7bd34c34afc455c900351bab6e1a8dc"
format_stamp: "Formatted at 2026-08-08 02:23:03 on dist-test-slave-zkpd"
server_key: "c2c40cbc05e1569f67b0835e50b8f8f0"
server_key_iv: "d9e1982cbbc4cc5b6334254128d15541"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.536296 8068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:04.546228 8068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:04.548246 8068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:04.548408 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:04.548622 8068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:04.548919 8068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:04.548981 8068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.549021 8068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:04.549049 8068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.554606 8068 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38063
I20260808 02:23:04.554661 8201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38063 every 8 connection(s)
I20260808 02:23:04.554970 8068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:23:04.559741 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8068
I20260808 02:23:04.560077 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:45375
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:45279
--webserver_port=45935
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.562656 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.555636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:42705)
0808 02:23:04.555806 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.555842 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.556368 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.556618 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.556622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.557163 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.557370 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.557374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.558073 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.558074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.558888 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.558890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.559283 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.559284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.559439 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.560088 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.560097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.562523 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
W20260808 02:23:04.562713 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.555657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:42705 (local address 127.30.160.62:45279)
0808 02:23:04.555702 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.555704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.555947 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:23:04.556475 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.556477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.556496 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.556551 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.557475 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.557966 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.559004 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.559197 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.559275 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.562643 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:42705: BlockingRecv error: recv got EOF from 127.30.160.2:42705 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 02:23:04.562932 8202 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:45279 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.160.62:45279: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.567831 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.563138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:40211)
0808 02:23:04.563287 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.563297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.563346 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.563481 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.563482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.563588 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.563630 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.563631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.564572 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.564573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.565080 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.565081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.565585 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.565586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.565631 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.566058 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.566062 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.567783 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
W20260808 02:23:04.568009 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.563209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:40211 (local address 127.30.160.62:45279)
0808 02:23:04.563266 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.563268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.563341 (+ 73us) server_negotiation.cc:407] Connection header received
0808 02:23:04.563406 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.563407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.563417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.563466 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.563698 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.564476 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.565161 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.565500 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.565572 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.567928 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:40211: BlockingRecv error: recv got EOF from 127.30.160.2:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.572566 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.568077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:51233)
0808 02:23:04.568162 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.568171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.568206 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.568396 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.568397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.568497 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.568521 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.568522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.569421 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.569421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.569871 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.569872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.570374 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.570375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.570409 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.570825 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.570829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.572527 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 02:23:04.572681 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.568154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:51233 (local address 127.30.160.62:45279)
0808 02:23:04.568276 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.568278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.568288 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:04.568311 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.568312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.568320 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.568387 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.568597 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.569343 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.569950 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.570283 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.570378 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.572627 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:51233: BlockingRecv error: recv got EOF from 127.30.160.2:51233 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260808 02:23:04.572721 8202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:23:04.645242 8205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:04.645442 8205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:04.645464 8205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:04.646607 8205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:04.646647 8205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:04.646698 8205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:23:04.648218 8205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/krb5kdc/oryx_127.30.160.3.keytab
--principal=oryx/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:45375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=45935
--tserver_master_addrs=127.30.160.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:04.648471 8205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:04.648699 8205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:04.653566 8211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.653684 8205 server_base.cc:1034] running on GCE node
W20260808 02:23:04.653609 8212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:04.653788 8214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:04.654006 8205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:04.654203 8205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:04.655368 8205 hybrid_clock.cc:648] HybridClock initialized: now 1786155784655343 us; error 35 us; skew 500 ppm
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155784, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:04.657821 8205 init.cc:377] Logged in from keytab as oryx/127.30.160.3@KRBTEST.COM (short username oryx)
I20260808 02:23:04.658243 8205 webserver.cc:460] Webserver started at http://127.30.160.3:45935/ using document root <none> and password file <none>
I20260808 02:23:04.658469 8205 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:04.658569 8205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:04.659875 8205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:23:04.660506 8222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.660693 8205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:23:04.660789 8205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "f1ac904608f048c4a846e81a29cd2374"
format_stamp: "Formatted at 2026-08-08 02:23:04 on dist-test-slave-zkpd"
server_key: "56fdecc6e7c3437eb0459911285d9c69"
server_key_iv: "add083e5d6ff38626ece8cb28fa5b356"
server_key_version: "encryptionkey@0"
I20260808 02:23:04.661154 8205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:04.673561 8205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:04.675284 8205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:04.675441 8205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:04.675653 8205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:04.676047 8205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:04.676102 8205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.676142 8205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:04.676169 8205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:04.681917 8205 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:45375
I20260808 02:23:04.681983 8339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:45375 every 8 connection(s)
I20260808 02:23:04.682353 8205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:23:04.686116 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8205
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:23:04.690047 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.683024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:49455 (local address 127.30.160.62:45279)
0808 02:23:04.683077 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.683079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.683263 (+ 184us) server_negotiation.cc:407] Connection header received
0808 02:23:04.683789 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.683791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.683803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.683862 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.684822 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.685323 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.686361 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.686620 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.686694 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.689981 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:49455: BlockingRecv error: recv got EOF from 127.30.160.3:49455 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:23:04.690091 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.682987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:49455)
0808 02:23:04.683129 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.683177 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.683677 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.683960 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.683964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.684521 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.684713 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.684718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.685424 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.685424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.686262 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.686265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.686721 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.686722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.686884 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.687393 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.687404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.689891 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260808 02:23:04.690315 8340 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:45279 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.160.62:45279: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.694058 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.686671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.30.160.62:45279)
0808 02:23:04.686726 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.686728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.686735 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:23:04.686789 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.686789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.686800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.686872 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.687642 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.688274 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.689283 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.689473 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.689523 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.690996 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:04.691000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:04.691000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:04.691009 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:04.692587 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:04.692974 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:04.692977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:04.692977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:04.693068 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:04.693265 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:04.693266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:04.693266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:04.693556 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:04.693692 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:04.693841 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:23:04.693966 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.694873 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.690539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:45627)
0808 02:23:04.690646 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.690658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.690714 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.690836 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.690837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.690975 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.691030 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.691031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.691644 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.691645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.692185 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.692187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.692524 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.692525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.692599 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.692933 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.692938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.694818 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260808 02:23:04.694986 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.690595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:45627 (local address 127.30.160.62:45279)
0808 02:23:04.690647 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.690648 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.690698 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:23:04.690776 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.690776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.690784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.690831 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.691117 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.691549 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.692273 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.692457 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.692525 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.694901 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:45627: BlockingRecv error: recv got EOF from 127.30.160.3:45627 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:04 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:04.698561 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:04.695151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:60053)
0808 02:23:04.695187 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:04.695199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:04.695242 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:04.695370 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:04.695371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:04.695469 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:04.695493 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.695494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.696108 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.696108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.696569 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:04.696570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.696856 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.696857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:04.696891 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.697180 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:04.697184 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:04.698522 (+ 1338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260808 02:23:04.698684 8067 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:04.695212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:60053 (local address 127.30.160.62:45279)
0808 02:23:04.695273 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:04.695275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:04.695281 (+ 6us) server_negotiation.cc:407] Connection header received
0808 02:23:04.695297 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:04.695298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:04.695310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:04.695365 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:04.695549 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.696046 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.696633 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:04.696796 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:04.696843 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:04.698616 (+ 1773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:60053: BlockingRecv error: recv got EOF from 127.30.160.3:60053 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 02:23:04.698680 8340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 02:23:05 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:05 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:05.458040 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:05.453548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:34275)
0808 02:23:05.453711 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:05.453721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:05.453770 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:05.453912 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:05.453913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:05.454072 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:05.454148 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:05.454150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.454797 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.454798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:05.455337 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:05.455338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.455725 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.455726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:05.455775 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:05.456276 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:05.456280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:05.457982 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260808 02:23:05.458215 8345 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:05.453631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:34275 (local address 127.30.160.62:45279)
0808 02:23:05.453771 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:05.453773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:05.453804 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:23:05.453829 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:05.453829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:05.453840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:05.453890 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:05.454217 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.454730 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.455409 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.455661 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.455711 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:05.458097 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:34275: BlockingRecv error: recv got EOF from 127.30.160.1:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
Aug 08 02:23:05 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:05 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:05.578145 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:05.573209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:59819)
0808 02:23:05.573419 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:05.573434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:05.573515 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:05.573680 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:05.573682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:05.573896 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:05.573969 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:05.573971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.574664 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.574665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:05.575224 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:05.575226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.575609 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.575611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:05.575681 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:05.576259 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:05.576264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:05.578043 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
W20260808 02:23:05.578197 8345 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:05.573357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:59819 (local address 127.30.160.62:45279)
0808 02:23:05.573434 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:05.573436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:05.573494 (+ 58us) server_negotiation.cc:407] Connection header received
0808 02:23:05.573580 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:05.573581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:05.573601 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:05.573671 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:05.574057 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.574566 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.575326 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.575528 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.575612 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:05.578131 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:59819: BlockingRecv error: recv got EOF from 127.30.160.2:59819 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 02:23:05 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:05 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:05.703702 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:05.699112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:52015)
0808 02:23:05.699237 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:05.699247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:05.699300 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:05.699416 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:05.699417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:05.699582 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:05.699632 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:05.699634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.700384 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.700385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:05.700948 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:05.700950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.701278 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.701279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:05.701339 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:05.701852 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:05.701857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:05.703637 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260808 02:23:05.703842 8345 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:05.699144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:52015 (local address 127.30.160.62:45279)
0808 02:23:05.699188 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:05.699190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:05.699286 (+ 96us) server_negotiation.cc:407] Connection header received
0808 02:23:05.699356 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:05.699357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:05.699370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:05.699404 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:05.699691 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.700293 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.701025 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:05.701203 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:05.701252 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:05.703730 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:52015: BlockingRecv error: recv got EOF from 127.30.160.3:52015 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 08 02:23:06 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:06 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:06.463357 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:06.458739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:43641)
0808 02:23:06.458900 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:06.458916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:06.458965 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:06.459100 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:06.459100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:06.459257 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:06.459312 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:06.459314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.460111 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.460112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:06.460629 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:06.460630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.461007 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.461008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:06.461047 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:06.461564 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:06.461569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:06.463296 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
W20260808 02:23:06.463506 8349 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:06.458815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:43641 (local address 127.30.160.62:45279)
0808 02:23:06.458968 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:06.458970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:06.459000 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:23:06.459022 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:06.459022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:06.459033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:06.459080 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:06.459417 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.460039 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.460718 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.460937 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.460993 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:06.463417 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:43641: BlockingRecv error: recv got EOF from 127.30.160.1:43641 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
Aug 08 02:23:06 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:06 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:06.583600 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:06.578675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:51689)
0808 02:23:06.578874 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:06.578884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:06.578943 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:06.579073 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:06.579073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:06.579254 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:06.579304 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:06.579306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.579972 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.579973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:06.580529 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:06.580531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.580868 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.580869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:06.580936 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:06.581449 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:06.581454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:06.583541 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":133,"threads_started":1}
W20260808 02:23:06.583741 8349 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:06.578748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:51689 (local address 127.30.160.62:45279)
0808 02:23:06.578799 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:06.578801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:06.578946 (+ 145us) server_negotiation.cc:407] Connection header received
0808 02:23:06.578986 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:06.578987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:06.578997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:06.579050 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:06.579369 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.579852 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.580599 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.580784 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.580863 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:06.583655 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:51689: BlockingRecv error: recv got EOF from 127.30.160.2:51689 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 08 02:23:06 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:06 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:06.709220 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:06.704241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:44287)
0808 02:23:06.704445 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:06.704457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:06.704544 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:06.704819 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:06.704820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:06.705044 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:06.705110 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:06.705112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.705761 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.705762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:06.706280 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:06.706281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.706605 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.706606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:06.706647 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:06.707175 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:06.707181 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:06.709118 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":127,"threads_started":1}
W20260808 02:23:06.709348 8349 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:06.704572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:44287 (local address 127.30.160.62:45279)
0808 02:23:06.704650 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:06.704652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:06.704663 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:23:06.704690 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:06.704691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:06.704706 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:06.704768 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:06.705202 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.705693 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.706367 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:06.706541 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:06.706591 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:06.709251 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:44287: BlockingRecv error: recv got EOF from 127.30.160.3:44287 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 02:23:07 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:07 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:07.468915 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:07.463835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:33207)
0808 02:23:07.464142 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:07.464154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:07.464237 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:07.464412 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:07.464413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:07.464636 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:07.464700 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:07.464703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.465364 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.465365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:07.465908 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:07.465909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.466421 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.466423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:07.466487 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:07.466998 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:07.467004 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:07.468846 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":102,"threads_started":1}
W20260808 02:23:07.469026 8354 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:07.463868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:33207 (local address 127.30.160.62:45279)
0808 02:23:07.464022 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:07.464024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:07.464233 (+ 209us) server_negotiation.cc:407] Connection header received
0808 02:23:07.464307 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:07.464308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:07.464322 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:07.464381 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:07.464781 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.465288 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.465988 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.466337 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.466409 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:07.468945 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:33207: BlockingRecv error: recv got EOF from 127.30.160.1:33207 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
Aug 08 02:23:07 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:07 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:07.588742 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:07.584118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:47679)
0808 02:23:07.584285 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:07.584303 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:07.584424 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:07.584585 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:07.584586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:07.584780 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:07.584836 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:07.584837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.585477 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.585477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:07.586010 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:07.586011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.586334 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.586335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:07.586389 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:07.586827 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:07.586832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:07.588671 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
W20260808 02:23:07.588848 8354 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:07.584166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:47679 (local address 127.30.160.62:45279)
0808 02:23:07.584213 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:07.584216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:07.584423 (+ 207us) server_negotiation.cc:407] Connection header received
0808 02:23:07.584497 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:07.584498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:07.584513 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:07.584562 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:07.584903 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.585394 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.586077 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.586249 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.586298 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:07.588760 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:47679: BlockingRecv error: recv got EOF from 127.30.160.2:47679 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:23:07 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:07 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:07.715143 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:07.709689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:58341)
0808 02:23:07.709901 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:07.709915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:07.709994 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:07.710131 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:07.710132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:07.710321 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:07.710373 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:07.710375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.711053 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.711054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:07.711688 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:07.711690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.712145 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.712147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:07.712215 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:07.712872 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:07.712877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:07.715064 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":128,"threads_started":1}
W20260808 02:23:07.715188 8354 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:07.709781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:58341 (local address 127.30.160.62:45279)
0808 02:23:07.709878 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:07.709880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:07.709993 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:23:07.710045 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:07.710046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:07.710058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:07.710108 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:07.710436 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.710933 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.711762 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:07.712057 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:07.712125 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:07.715137 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:58341: BlockingRecv error: recv got EOF from 127.30.160.3:58341 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 02:23:08 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:08 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:08.473853 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:08.469383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:33781)
0808 02:23:08.469546 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:08.469556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:08.469612 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:08.469740 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:08.469741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:08.469910 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:08.469961 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:08.469962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.470596 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.470597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:08.471123 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:08.471124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.471448 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.471449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:08.471488 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:08.472039 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:08.472045 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:08.473787 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
W20260808 02:23:08.474011 8358 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:08.469476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:33781 (local address 127.30.160.62:45279)
0808 02:23:08.469593 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:08.469595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:08.469604 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:23:08.469659 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:08.469660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:08.469670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:08.469725 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:08.470036 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.470531 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.471196 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.471386 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.471434 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:08.473918 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:33781: BlockingRecv error: recv got EOF from 127.30.160.1:33781 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
Aug 08 02:23:08 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:08 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:08.597985 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:08.589245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:56901)
0808 02:23:08.589393 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:08.589405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:08.589473 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:08.589586 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:08.589586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:08.589751 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:08.589800 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:08.589801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.590430 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.590431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:08.590953 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:08.590954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.591269 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.591270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:08.591308 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:08.596115 (+ 4807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:08.596120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:08.597922 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
W20260808 02:23:08.598099 8358 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:08.589302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:56901 (local address 127.30.160.62:45279)
0808 02:23:08.589357 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:08.589359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:08.589457 (+ 98us) server_negotiation.cc:407] Connection header received
0808 02:23:08.589528 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:08.589529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:08.589540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:08.589579 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:08.589874 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.590361 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.591022 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.591205 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.591255 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:08.598028 (+ 6773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:56901: BlockingRecv error: recv got EOF from 127.30.160.2:56901 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 08 02:23:08 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:08 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:08.720237 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:08.715653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:53043)
0808 02:23:08.715815 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:08.715827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:08.715931 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:08.716077 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:08.716078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:08.716243 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:08.716294 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:08.716295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.716939 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.716940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:08.717464 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:08.717466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.717862 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.717863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:08.717935 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:08.718384 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:08.718389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:08.720176 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
W20260808 02:23:08.720335 8358 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:08.715707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:53043 (local address 127.30.160.62:45279)
0808 02:23:08.715756 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:08.715758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:08.715906 (+ 148us) server_negotiation.cc:407] Connection header received
0808 02:23:08.715995 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:08.715996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:08.716008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:08.716055 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:08.716367 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.716877 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.717544 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:08.717750 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:08.717833 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:08.720267 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:53043: BlockingRecv error: recv got EOF from 127.30.160.3:53043 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:23:09 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:09 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:09.479001 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:09.474330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:35547)
0808 02:23:09.474493 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:09.474507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:09.474607 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:09.474771 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:09.474772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:09.474991 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:09.475060 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:09.475063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.475748 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.475749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:09.476336 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:09.476337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.476664 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.476665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:09.476703 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:09.477144 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:09.477148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:09.478942 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
W20260808 02:23:09.479153 8363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:09.474379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:35547 (local address 127.30.160.62:45279)
0808 02:23:09.474543 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:09.474546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:09.474585 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:23:09.474665 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:09.474666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:09.474680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:09.474784 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:09.475143 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.475678 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.476403 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.476598 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.476655 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:09.479054 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:35547: BlockingRecv error: recv got EOF from 127.30.160.1:35547 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
Aug 08 02:23:09 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:09 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:09.603060 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:09.598508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:43291)
0808 02:23:09.598691 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:09.598702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:09.598760 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:09.598881 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:09.598881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:09.599047 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:09.599098 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:09.599100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.599729 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.599729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:09.600282 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:09.600283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.600642 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.600642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:09.600682 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:09.601169 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:09.601192 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:09.602955 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
W20260808 02:23:09.603158 8363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:09.598639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:43291 (local address 127.30.160.62:45279)
0808 02:23:09.598688 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:09.598690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:09.598772 (+ 82us) server_negotiation.cc:407] Connection header received
0808 02:23:09.598801 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:09.598802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:09.598812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:09.598860 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:09.599171 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.599664 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.600378 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.600568 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.600631 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:09.603075 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:43291: BlockingRecv error: recv got EOF from 127.30.160.2:43291 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 08 02:23:09 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:09 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:09.725669 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:09.720871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:55041)
0808 02:23:09.721042 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:09.721053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:09.721108 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:09.721240 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:09.721241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:09.721418 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:09.721470 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:09.721471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.722282 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.722283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:09.722855 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:09.722857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.723270 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.723271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:09.723339 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:09.723766 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:09.723770 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:09.725609 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
W20260808 02:23:09.725796 8363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:09.720973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:55041 (local address 127.30.160.62:45279)
0808 02:23:09.721058 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:09.721061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:09.721108 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:23:09.721152 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:09.721152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:09.721164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:09.721219 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:09.721536 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.722193 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.722993 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:09.723195 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:09.723272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:09.725694 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:55041: BlockingRecv error: recv got EOF from 127.30.160.3:55041 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:23:10 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:10 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:10.484513 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:10.479517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:49391)
0808 02:23:10.479676 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:10.479694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:10.479757 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:10.479906 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:10.479907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:10.480101 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:10.480150 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:10.480152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.480889 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.480890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:10.481492 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:10.481494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.481885 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.481887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:10.481945 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:10.482492 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:10.482498 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:10.484414 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
W20260808 02:23:10.484572 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:10.479569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:49391 (local address 127.30.160.62:45279)
0808 02:23:10.479709 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:10.479711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:10.479740 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:23:10.479810 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:10.479810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:10.479826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:10.479934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:10.480214 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.480800 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.481569 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.481767 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.481867 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:10.484497 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:49391: BlockingRecv error: recv got EOF from 127.30.160.1:49391 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
Aug 08 02:23:10 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:10 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:10.608340 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:10.603524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:57511)
0808 02:23:10.603654 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:10.603665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:10.603730 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:10.603871 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:10.603872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:10.604144 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:10.604206 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:10.604208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.605015 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.605016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:10.605563 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:10.605564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.605980 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.605980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:10.606023 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:10.606471 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:10.606475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:10.608277 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
W20260808 02:23:10.608476 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:10.603611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:57511 (local address 127.30.160.62:45279)
0808 02:23:10.603665 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:10.603667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:10.603735 (+ 68us) server_negotiation.cc:407] Connection header received
0808 02:23:10.603779 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:10.603780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:10.603792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:10.603854 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:10.604288 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.604939 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.605634 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.605897 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.605968 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:10.608392 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:57511: BlockingRecv error: recv got EOF from 127.30.160.2:57511 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:23:10 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:10 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:10.730861 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:10.726156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:43221)
0808 02:23:10.726332 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:10.726343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:10.726400 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:10.726542 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:10.726543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:10.726711 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:10.726775 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:10.726777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.727406 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.727407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:10.727974 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:10.727975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.728296 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.728297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:10.728348 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:10.728775 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:10.728779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:10.730757 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
W20260808 02:23:10.730903 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:10.726287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:43221 (local address 127.30.160.62:45279)
0808 02:23:10.726369 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:10.726371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:10.726384 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:23:10.726462 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:10.726462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:10.726473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:10.726522 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:10.726844 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.727336 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.728045 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:10.728230 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:10.728283 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:10.730838 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:43221: BlockingRecv error: recv got EOF from 127.30.160.3:43221 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:23:11 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:11 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:11.489706 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:11.484982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:33713)
0808 02:23:11.485140 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:11.485151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:11.485209 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:11.485359 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:11.485360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:11.485523 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:11.485575 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:11.485577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.486215 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.486215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:11.486936 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:11.486940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.487360 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.487361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:11.487404 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:11.487821 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:11.487825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:11.489646 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
W20260808 02:23:11.489826 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:11.485067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:33713 (local address 127.30.160.62:45279)
0808 02:23:11.485186 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:11.485188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:11.485197 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:23:11.485268 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:11.485268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:11.485280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:11.485341 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:11.485635 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.486135 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.487043 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.487291 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.487351 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:11.489743 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:33713: BlockingRecv error: recv got EOF from 127.30.160.1:33713 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
Aug 08 02:23:11 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:11 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:11.613722 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:11.608792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:54643)
0808 02:23:11.609010 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:11.609022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:11.609095 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:11.609260 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:11.609262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:11.609466 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:11.609516 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:11.609517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.610202 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.610203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:11.610744 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:11.610746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.611121 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.611122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:11.611184 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:11.611716 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:11.611722 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:11.613612 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":125,"threads_started":1}
W20260808 02:23:11.613776 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:11.608917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:54643 (local address 127.30.160.62:45279)
0808 02:23:11.608994 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:11.608997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:11.609092 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:23:11.609155 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:11.609155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:11.609169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:11.609237 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:11.609599 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.610114 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.610836 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.611033 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.611120 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:11.613708 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:54643: BlockingRecv error: recv got EOF from 127.30.160.2:54643 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 02:23:11 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:11 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:11.736215 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:11.731415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:55523)
0808 02:23:11.731567 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:11.731581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:11.731644 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:11.731777 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:11.731777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:11.732020 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:11.732072 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:11.732074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.732760 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.732762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:11.733455 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:11.733458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.733797 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.733798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:11.733839 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:11.734285 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:11.734290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:11.736156 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
W20260808 02:23:11.736392 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:11.731507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:55523 (local address 127.30.160.62:45279)
0808 02:23:11.731572 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:11.731578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:11.731645 (+ 67us) server_negotiation.cc:407] Connection header received
0808 02:23:11.731701 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:11.731702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:11.731712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:11.731754 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:11.732138 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.732668 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.733532 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:11.733728 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:11.733782 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:11.736313 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:55523: BlockingRecv error: recv got EOF from 127.30.160.3:55523 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 08 02:23:12 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:12 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:12.494835 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:12.490291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:45125)
0808 02:23:12.490463 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:12.490474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:12.490527 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:12.490671 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:12.490672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:12.490839 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:12.490891 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:12.490893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.491534 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.491535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:12.492112 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:12.492114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.492440 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.492440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:12.492480 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:12.492927 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:12.492932 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:12.494737 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
W20260808 02:23:12.494901 8375 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:12.490428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:45125 (local address 127.30.160.62:45279)
0808 02:23:12.490552 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:12.490559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:12.490567 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:23:12.490589 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:12.490589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:12.490600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:12.490657 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:12.490959 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.491469 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.492180 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.492377 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.492427 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:12.494831 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:45125: BlockingRecv error: recv got EOF from 127.30.160.1:45125 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
Aug 08 02:23:12 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:12 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:12.618999 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:12.614218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:59609)
0808 02:23:12.614394 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:12.614404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:12.614462 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:12.614576 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:12.614576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:12.614740 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:12.614792 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:12.614794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.615636 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.615637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:12.616190 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:12.616191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.616521 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.616522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:12.616562 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:12.616985 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:12.616989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:12.618937 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
W20260808 02:23:12.619099 8375 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:12.614261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:59609 (local address 127.30.160.62:45279)
0808 02:23:12.614337 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:12.614339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:12.614462 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:23:12.614519 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:12.614519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:12.614530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:12.614573 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:12.614879 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.615551 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.616259 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.616452 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.616507 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:12.619029 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:59609: BlockingRecv error: recv got EOF from 127.30.160.2:59609 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 08 02:23:12 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:12 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:12.741422 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:12.736676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:34575)
0808 02:23:12.736857 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:12.736870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:12.736961 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:12.737108 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:12.737108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:12.737291 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:12.737346 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:12.737348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.738080 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.738081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:12.738645 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:12.738647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.739036 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.739037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:12.739102 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:12.739512 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:12.739516 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:12.741309 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
W20260808 02:23:12.741463 8375 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:12.736767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:34575 (local address 127.30.160.62:45279)
0808 02:23:12.736852 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:12.736854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:12.736939 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:23:12.737035 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:12.737036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:12.737051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:12.737104 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:12.737426 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.737992 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.738731 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:12.738943 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:12.739024 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:12.741400 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:34575: BlockingRecv error: recv got EOF from 127.30.160.3:34575 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 02:23:13 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:13 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:13.500082 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:13.495331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:50495)
0808 02:23:13.495468 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:13.495477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:13.495528 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:13.495649 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:13.495649 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:13.495803 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:13.495860 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:13.495862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.496596 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.496597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:13.497137 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:13.497138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.497617 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.497618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:13.497685 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:13.498111 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:13.498116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:13.500021 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
W20260808 02:23:13.500204 8379 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:13.495384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:50495 (local address 127.30.160.62:45279)
0808 02:23:13.495510 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:13.495512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:13.495523 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:23:13.495584 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:13.495585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:13.495595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:13.495644 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:13.495999 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.496517 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.497231 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.497536 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.497617 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:13.500120 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:50495: BlockingRecv error: recv got EOF from 127.30.160.1:50495 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
Aug 08 02:23:13 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:13 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:13.628736 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:13.619495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:37183)
0808 02:23:13.619656 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:13.619665 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:13.619722 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:13.619868 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:13.619869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:13.620082 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:13.620136 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:13.620138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.620760 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.620761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:13.621287 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:13.621289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.621625 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.621626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:13.621668 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:13.626702 (+ 5034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:13.626708 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:13.628674 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
W20260808 02:23:13.628885 8379 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:13.619573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:37183 (local address 127.30.160.62:45279)
0808 02:23:13.619652 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:13.619653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:13.619724 (+ 71us) server_negotiation.cc:407] Connection header received
0808 02:23:13.619767 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:13.619767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:13.619778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:13.619839 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:13.620209 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.620692 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.621371 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.621559 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.621612 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:13.628799 (+ 7187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:37183: BlockingRecv error: recv got EOF from 127.30.160.2:37183 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 02:23:13 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:13 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:13.750749 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:13.741889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:39249)
0808 02:23:13.742064 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:13.742077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:13.742179 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:13.742373 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:13.742374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:13.742556 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:13.742604 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:13.742606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.743283 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.743284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:13.743854 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:13.743855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.744279 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.744280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:13.744349 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:13.748752 (+ 4403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:13.748758 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:13.750648 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
W20260808 02:23:13.750799 8379 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:13.741982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:39249 (local address 127.30.160.62:45279)
0808 02:23:13.742061 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:13.742064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:13.742161 (+ 97us) server_negotiation.cc:407] Connection header received
0808 02:23:13.742266 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:13.742267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:13.742280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:13.742364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:13.742685 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.743180 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.743985 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:13.744189 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:13.744276 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:13.750730 (+ 6454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:39249: BlockingRecv error: recv got EOF from 127.30.160.3:39249 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 02:23:14 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:14 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:14.505182 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:14.500568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:39759)
0808 02:23:14.500716 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:14.500730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:14.500794 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:14.500940 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:14.500941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:14.501110 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:14.501159 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:14.501161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.501833 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.501834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:14.502352 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:14.502353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.502676 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.502677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:14.502716 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:14.503186 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:14.503190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:14.505085 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260808 02:23:14.505229 8383 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:14.500601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:39759 (local address 127.30.160.62:45279)
0808 02:23:14.500759 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:14.500761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:14.500776 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:23:14.500854 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:14.500855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:14.500866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:14.500920 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:14.501235 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.501763 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.502422 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.502614 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.502663 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:14.505160 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:39759: BlockingRecv error: recv got EOF from 127.30.160.1:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":49,"threads_started":1}
Aug 08 02:23:14 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:14 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:14.634138 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:14.629221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:44357)
0808 02:23:14.629455 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:14.629468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:14.629546 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:14.629746 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:14.629747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:14.629931 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:14.629984 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:14.629986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.630673 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.630674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:14.631246 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:14.631248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.631610 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.631611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:14.631665 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:14.632249 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:14.632254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:14.634078 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":125,"threads_started":1}
W20260808 02:23:14.634261 8383 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:14.629329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:44357 (local address 127.30.160.62:45279)
0808 02:23:14.629399 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:14.629402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:14.629538 (+ 136us) server_negotiation.cc:407] Connection header received
0808 02:23:14.629631 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:14.629633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:14.629648 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:14.629710 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:14.630090 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.630596 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.631334 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.631520 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.631576 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:14.634181 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:44357: BlockingRecv error: recv got EOF from 127.30.160.2:44357 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 02:23:14 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:14 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:14.756142 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:14.751324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:41623)
0808 02:23:14.751474 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:14.751486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:14.751537 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:14.751674 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:14.751675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:14.751911 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:14.751967 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:14.751970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.752613 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.752614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:14.753218 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:14.753220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.753660 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.753661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:14.753729 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:14.754271 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:14.754276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:14.756072 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260808 02:23:14.756227 8383 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:14.751455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:41623 (local address 127.30.160.62:45279)
0808 02:23:14.751530 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:14.751532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:14.751540 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:23:14.751592 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:14.751593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:14.751605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:14.751654 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:14.752040 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.752536 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.753307 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:14.753485 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:14.753533 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:14.756162 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:41623: BlockingRecv error: recv got EOF from 127.30.160.3:41623 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 02:23:15 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:15 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:15.510373 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:15.505714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:46813)
0808 02:23:15.505884 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:15.505895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:15.505951 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:15.506138 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:15.506139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:15.506308 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:15.506373 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:15.506374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.506995 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.506996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:15.507525 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:15.507526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.507847 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.507848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:15.507887 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:15.508331 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:15.508335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:15.510286 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
W20260808 02:23:15.510442 8387 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:15.505869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:46813 (local address 127.30.160.62:45279)
0808 02:23:15.506019 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:15.506021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:15.506028 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:23:15.506051 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:15.506051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:15.506062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:15.506120 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:15.506441 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.506930 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.507594 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.507780 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.507829 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:15.510388 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:46813: BlockingRecv error: recv got EOF from 127.30.160.1:46813 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
Aug 08 02:23:15 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:15 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:15.639508 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:15.634704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:57617)
0808 02:23:15.634873 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:15.634884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:15.634948 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:15.635088 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:15.635089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:15.635262 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:15.635313 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:15.635314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.635951 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.635951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:15.636468 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:15.636469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.636801 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.636801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:15.636891 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:15.637501 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:15.637506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:15.639373 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
W20260808 02:23:15.639546 8387 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:15.634760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:57617 (local address 127.30.160.62:45279)
0808 02:23:15.634838 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:15.634841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:15.634949 (+ 108us) server_negotiation.cc:407] Connection header received
0808 02:23:15.635013 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:15.635014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:15.635026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:15.635068 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:15.635377 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.635874 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.636534 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.636718 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.636797 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:15.639472 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:57617: BlockingRecv error: recv got EOF from 127.30.160.2:57617 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 02:23:15 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:15 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:15.761097 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:15.756610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:34333)
0808 02:23:15.756753 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:15.756762 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:15.756825 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:15.756981 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:15.756982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:15.757148 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:15.757198 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:15.757200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.757824 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.757825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:15.758361 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:15.758362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.758663 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.758664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:15.758705 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:15.759148 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:15.759152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:15.761027 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260808 02:23:15.761216 8387 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:15.756790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:34333 (local address 127.30.160.62:45279)
0808 02:23:15.756860 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:15.756862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:15.756870 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:23:15.756898 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:15.756899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:15.756910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:15.756961 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:15.757259 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.757748 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.758427 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:15.758600 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:15.758649 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:15.761115 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:34333: BlockingRecv error: recv got EOF from 127.30.160.3:34333 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 02:23:16 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:16 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:16.516628 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:16.510887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:36411)
0808 02:23:16.511060 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:16.511074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:16.511132 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:16.511279 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:16.511279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:16.511457 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:16.511507 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:16.511509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.512608 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.512609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:16.513192 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:16.513194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.513797 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.513798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:16.513847 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:16.514390 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:16.514395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:16.516505 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
W20260808 02:23:16.516649 8392 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:16.510947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:36411 (local address 127.30.160.62:45279)
0808 02:23:16.511125 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:16.511128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:16.511140 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:23:16.511176 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:16.511177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:16.511191 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:16.511268 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:16.511586 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.512524 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.513272 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.513720 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.513797 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:16.516594 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:36411: BlockingRecv error: recv got EOF from 127.30.160.1:36411 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
Aug 08 02:23:16 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:16 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:16.644371 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:16.639986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:40415)
0808 02:23:16.640144 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:16.640156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:16.640227 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:16.640365 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:16.640366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:16.640540 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:16.640590 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:16.640592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.641207 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.641207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:16.641743 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:16.641744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.642061 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.642062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:16.642105 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:16.642524 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:16.642528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:16.644298 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
W20260808 02:23:16.644470 8392 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:16.640034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:40415 (local address 127.30.160.62:45279)
0808 02:23:16.640079 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:16.640081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:16.640227 (+ 146us) server_negotiation.cc:407] Connection header received
0808 02:23:16.640285 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:16.640286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:16.640297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:16.640344 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:16.640654 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.641140 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.641815 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.641996 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.642046 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:16.644402 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:40415: BlockingRecv error: recv got EOF from 127.30.160.2:40415 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:23:16 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:16 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:16.765986 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:16.761544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:45009)
0808 02:23:16.761728 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:16.761739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:16.761797 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:16.761940 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:16.761941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:16.762113 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:16.762161 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:16.762162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.762784 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.762785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:16.763311 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:16.763313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.763622 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.763623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:16.763681 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:16.764185 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:16.764189 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:16.765925 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
W20260808 02:23:16.766094 8392 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:16.761624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:45009 (local address 127.30.160.62:45279)
0808 02:23:16.761681 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:16.761683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:16.761794 (+ 111us) server_negotiation.cc:407] Connection header received
0808 02:23:16.761857 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:16.761857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:16.761868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:16.761922 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:16.762222 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.762716 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.763381 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:16.763558 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:16.763607 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:16.766020 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:45009: BlockingRecv error: recv got EOF from 127.30.160.3:45009 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 08 02:23:17 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:17 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:17.521965 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:17.517255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:46441)
0808 02:23:17.517431 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:17.517442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:17.517533 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:17.517716 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:17.517717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:17.517919 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:17.517968 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:17.517970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.518671 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.518672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:17.519203 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:17.519205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.519588 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.519589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:17.519656 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:17.520127 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:17.520132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:17.521864 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
W20260808 02:23:17.522025 8396 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:17.517324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:46441 (local address 127.30.160.62:45279)
0808 02:23:17.517435 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:17.517437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:17.517513 (+ 76us) server_negotiation.cc:407] Connection header received
0808 02:23:17.517609 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:17.517610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:17.517628 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:17.517709 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:17.518070 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.518582 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.519292 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.519495 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.519585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:17.521950 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:46441: BlockingRecv error: recv got EOF from 127.30.160.1:46441 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
Aug 08 02:23:17 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:17 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:17.649272 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:17.644976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:36049)
0808 02:23:17.645157 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:17.645169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:17.645228 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:17.645372 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:17.645372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:17.645544 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:17.645595 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:17.645596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.646209 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.646210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:17.646719 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:17.646720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.647033 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.647034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:17.647074 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:17.647507 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:17.647512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:17.649213 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
W20260808 02:23:17.649379 8396 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:17.645101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:36049 (local address 127.30.160.62:45279)
0808 02:23:17.645172 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:17.645174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:17.645224 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:23:17.645285 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:17.645286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:17.645298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:17.645352 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:17.645658 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.646146 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.646787 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.646972 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.647021 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:17.649303 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:36049: BlockingRecv error: recv got EOF from 127.30.160.2:36049 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 02:23:17 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:17 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:17.770902 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:17.766492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:51845)
0808 02:23:17.766670 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:17.766681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:17.766742 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:17.766874 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:17.766875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:17.767047 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:17.767095 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:17.767096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.767727 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.767728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:17.768280 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:17.768281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.768605 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.768606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:17.768648 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:17.769096 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:17.769101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:17.770833 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":116,"threads_started":1}
W20260808 02:23:17.771026 8396 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:17.766543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:51845 (local address 127.30.160.62:45279)
0808 02:23:17.766613 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:17.766616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:17.766746 (+ 130us) server_negotiation.cc:407] Connection header received
0808 02:23:17.766795 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:17.766796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:17.766809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:17.766855 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:17.767158 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.767662 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.768356 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:17.768538 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:17.768591 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:17.770926 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:51845: BlockingRecv error: recv got EOF from 127.30.160.3:51845 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 02:23:18 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:18 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:18.526919 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:18.522461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:52187)
0808 02:23:18.522612 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:18.522629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:18.522681 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:18.522846 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:18.522846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:18.523018 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:18.523068 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:18.523069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.523708 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.523709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:18.524301 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:18.524304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.524645 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.524645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:18.524685 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:18.525147 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:18.525151 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:18.526861 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
W20260808 02:23:18.527043 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:18.522514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:52187 (local address 127.30.160.62:45279)
0808 02:23:18.522649 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:18.522651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:18.522663 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:23:18.522741 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:18.522742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:18.522752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:18.522813 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:18.523132 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.523644 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.524381 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.524575 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.524633 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:18.526957 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:52187: BlockingRecv error: recv got EOF from 127.30.160.1:52187 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
Aug 08 02:23:18 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:18 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:18.658622 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:18.649772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:46331)
0808 02:23:18.649941 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:18.649951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:18.650006 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:18.650142 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:18.650143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:18.650315 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:18.650367 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:18.650368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.650985 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.650986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:18.651499 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:18.651500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.651819 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.651821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:18.651861 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:18.656509 (+ 4648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:18.656514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:18.658524 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
W20260808 02:23:18.658674 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:18.649835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:46331 (local address 127.30.160.62:45279)
0808 02:23:18.649902 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:18.649904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:18.650007 (+ 103us) server_negotiation.cc:407] Connection header received
0808 02:23:18.650066 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:18.650067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:18.650078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:18.650119 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:18.650430 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.650920 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.651568 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.651749 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.651799 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:18.658607 (+ 6808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:46331: BlockingRecv error: recv got EOF from 127.30.160.2:46331 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 02:23:18 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:18 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.3@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:18.776175 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:18.771369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.3:57957)
0808 02:23:18.771565 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:18.771578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:18.771661 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:18.771810 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:18.771810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:18.772027 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:18.772078 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:18.772080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.772763 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.772765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:18.773305 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:18.773306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.773673 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.773674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:18.773798 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:18.774245 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:18.774249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:18.776103 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
W20260808 02:23:18.776286 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:18.771480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:57957 (local address 127.30.160.62:45279)
0808 02:23:18.771563 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:18.771565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:18.771640 (+ 75us) server_negotiation.cc:407] Connection header received
0808 02:23:18.771732 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:18.771733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:18.771749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:18.771795 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:18.772163 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.772672 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.773391 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:18.773585 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:18.773669 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:18.776190 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.3:57957: BlockingRecv error: recv got EOF from 127.30.160.3:57957 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 02:23:19 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:19 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.1@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:19.532661 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:19.527621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.1:54731)
0808 02:23:19.527799 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:19.527811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:19.527873 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:19.528187 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:19.528187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:19.528354 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:19.528408 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:19.528409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.529044 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.529045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:19.529551 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:19.529552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.529906 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.529907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:19.529960 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:19.530417 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:19.530421 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:19.532599 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
W20260808 02:23:19.532773 8404 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:19.527745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:54731 (local address 127.30.160.62:45279)
0808 02:23:19.527880 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:19.527882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:19.527916 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:23:19.528093 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:19.528094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:19.528104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:19.528164 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:19.528474 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.528977 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.529622 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.529809 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.529860 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:19.532694 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.1:54731: BlockingRecv error: recv got EOF from 127.30.160.1:54731 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":97,"threads_started":1}
Aug 08 02:23:19 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:23:19 dist-test-slave-zkpd krb5kdc[7312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.160.2@KRBTEST.COM for oryx/127.30.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:23:19.664225 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:19.659401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:45279 (local address 127.30.160.2:42929)
0808 02:23:19.659596 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:19.659606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:19.659689 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:19.659877 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:19.659878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:19.660082 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:19.660131 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:19.660133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.660827 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.660828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:19.661357 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:19.661358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.661807 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.661808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:19.661884 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:19.662303 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:19.662307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:19.664109 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.160.62:45279: Server oryx/127.30.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
W20260808 02:23:19.664237 8404 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:19.659465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:42929 (local address 127.30.160.62:45279)
0808 02:23:19.659556 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:19.659558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:19.659669 (+ 111us) server_negotiation.cc:407] Connection header received
0808 02:23:19.659769 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:19.659770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:19.659783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:19.659861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:19.660217 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.660728 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.661446 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.661731 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.661807 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:19.664185 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.160.2:42929: BlockingRecv error: recv got EOF from 127.30.160.2:42929 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:23:19.693634 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7914
I20260808 02:23:19.700500 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8068
I20260808 02:23:19.706149 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8205
I20260808 02:23:19.711760 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 7843
2026-08-08T02:23:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17495 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T02:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:23:19Z Disabled control of system clock
I20260808 02:23:19.741674 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44479
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:44479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:23:19.823503 8409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:19.823719 8409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:19.823738 8409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:19.824935 8409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:23:19.824980 8409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:19.824994 8409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:19.825007 8409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:23:19.825019 8409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:23:19.826308 8409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:44479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:19.826506 8409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:19.826737 8409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:19.831735 8417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:19.831724 8414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:19.831720 8409 server_base.cc:1034] running on GCE node
W20260808 02:23:19.831786 8415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:19.832234 8409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:19.832541 8409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:19.833709 8409 hybrid_clock.cc:648] HybridClock initialized: now 1786155799833683 us; error 51 us; skew 500 ppm
I20260808 02:23:19.834877 8409 webserver.cc:460] Webserver started at http://127.30.160.62:45627/ using document root <none> and password file <none>
I20260808 02:23:19.835117 8409 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:19.835186 8409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:19.835340 8409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:19.836539 8409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "454fd1d7596143668d9ec268c086647a"
format_stamp: "Formatted at 2026-08-08 02:23:19 on dist-test-slave-zkpd"
server_key: "5578b32f6e8d1a070e62b5a7cc183e5d"
server_key_iv: "957edcb71243aac8286557b17ab15ceb"
server_key_version: "encryptionkey@0"
I20260808 02:23:19.836905 8409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "454fd1d7596143668d9ec268c086647a"
format_stamp: "Formatted at 2026-08-08 02:23:19 on dist-test-slave-zkpd"
server_key: "5578b32f6e8d1a070e62b5a7cc183e5d"
server_key_iv: "957edcb71243aac8286557b17ab15ceb"
server_key_version: "encryptionkey@0"
I20260808 02:23:19.838202 8409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:23:19.838907 8423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:19.839076 8409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:19.839181 8409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "454fd1d7596143668d9ec268c086647a"
format_stamp: "Formatted at 2026-08-08 02:23:19 on dist-test-slave-zkpd"
server_key: "5578b32f6e8d1a070e62b5a7cc183e5d"
server_key_iv: "957edcb71243aac8286557b17ab15ceb"
server_key_version: "encryptionkey@0"
I20260808 02:23:19.839257 8409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:19.858342 8409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:19.858639 8409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:19.858793 8409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:19.863006 8409 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:44479
I20260808 02:23:19.863040 8478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:44479 every 8 connection(s)
I20260808 02:23:19.863373 8409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:23:19.864122 8479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:19.865940 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8409
I20260808 02:23:19.866046 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:23:19.866241 31360 external_mini_cluster.cc:1428] Setting key 7f52990544a7302d24489f8de6321477
I20260808 02:23:19.867183 8479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a: Bootstrap starting.
I20260808 02:23:19.867941 8479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:19.868250 8479 log.cc:824] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:19.868909 8479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a: No bootstrap required, opened a new log
I20260808 02:23:19.870662 8479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454fd1d7596143668d9ec268c086647a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44479 } }
I20260808 02:23:19.870841 8479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:19.870880 8479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454fd1d7596143668d9ec268c086647a, State: Initialized, Role: FOLLOWER
I20260808 02:23:19.871047 8479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454fd1d7596143668d9ec268c086647a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44479 } }
I20260808 02:23:19.871168 8479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:23:19.871208 8479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:23:19.871268 8479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:19.871805 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:19.866770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.30.160.62:44479)
0808 02:23:19.866985 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:19.866994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:19.867025 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:23:19.867608 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:19.867617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:19.867694 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:19.867849 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:19.868095 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.869043 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.869819 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:19.870220 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:19.870385 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:19.871081 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:19.871084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:19.871084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:19.871300 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:19.871355 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:19.871370 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:19.871442 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:19.871511 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:23:19.871591 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260808 02:23:19.871968 8479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454fd1d7596143668d9ec268c086647a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44479 } }
I20260808 02:23:19.872093 8479 leader_election.cc:302] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 454fd1d7596143668d9ec268c086647a; no voters:
I20260808 02:23:19.872252 8479 leader_election.cc:288] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:23:19.872310 8484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:19.872499 8479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:23:19.872520 8484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [term 1 LEADER]: Becoming Leader. State: Replica: 454fd1d7596143668d9ec268c086647a, State: Running, Role: LEADER
I20260808 02:23:19.872704 8484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454fd1d7596143668d9ec268c086647a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44479 } }
I20260808 02:23:19.873159 8486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "454fd1d7596143668d9ec268c086647a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454fd1d7596143668d9ec268c086647a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44479 } } }
I20260808 02:23:19.873144 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 454fd1d7596143668d9ec268c086647a. Latest consensus state: current_term: 1 leader_uuid: "454fd1d7596143668d9ec268c086647a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454fd1d7596143668d9ec268c086647a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44479 } } }
I20260808 02:23:19.873380 8486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:19.873390 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:19.873572 8494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:23:19.874109 8494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:23:19.875695 8494 catalog_manager.cc:1347] Generated new cluster ID: f1aa6428ea7843a5bee8130678db5fdb
I20260808 02:23:19.875749 8494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:23:19.881747 8494 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:23:19.882195 8494 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:23:19.888501 8494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 454fd1d7596143668d9ec268c086647a: Generated new TSK 0
I20260808 02:23:19.888651 8494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:23:19.891947 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44479
--builtin_ntp_servers=127.30.160.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 02:23:19.991089 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:19.991319 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:19.991355 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:19.992558 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:19.992615 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:19.992655 8504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:23:19.993988 8504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:19.994254 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:19.994553 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:20.000389 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.000396 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.000406 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.000970 8504 server_base.cc:1034] running on GCE node
I20260808 02:23:20.001176 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.001444 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.002641 8504 hybrid_clock.cc:648] HybridClock initialized: now 1786155800002629 us; error 29 us; skew 500 ppm
I20260808 02:23:20.004161 8504 webserver.cc:460] Webserver started at http://127.30.160.1:45615/ using document root <none> and password file <none>
I20260808 02:23:20.004467 8504 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.004540 8504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.004681 8504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.006119 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "0ce2a63c0a4845f0ab589a0605234094"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "daa1a185ffa398d2125d2f199fc2ff29"
server_key_iv: "53af5cff8e84572a808e21533dc6c391"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.006554 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "0ce2a63c0a4845f0ab589a0605234094"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "daa1a185ffa398d2125d2f199fc2ff29"
server_key_iv: "53af5cff8e84572a808e21533dc6c391"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.008635 8504 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 02:23:20.009560 8519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.009810 8504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:20.009931 8504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "0ce2a63c0a4845f0ab589a0605234094"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "daa1a185ffa398d2125d2f199fc2ff29"
server_key_iv: "53af5cff8e84572a808e21533dc6c391"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.010021 8504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.026444 8504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.026774 8504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.026940 8504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.027163 8504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:20.027501 8504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:20.027556 8504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.027596 8504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:20.027623 8504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.033365 8504 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42821
I20260808 02:23:20.033403 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42821 every 8 connection(s)
I20260808 02:23:20.033771 8504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:23:20.038146 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8504
I20260808 02:23:20.038264 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:23:20.038470 31360 external_mini_cluster.cc:1428] Setting key f08b8bafd589b2f838770533b5e8d503
I20260808 02:23:20.039449 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.034513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:60179 (local address 127.30.160.62:44479)
0808 02:23:20.034595 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.034597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.034830 (+ 233us) server_negotiation.cc:407] Connection header received
0808 02:23:20.035322 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.035323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.035333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.035382 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.036173 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.036723 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.037793 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.038141 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.038224 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:20.039170 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.039172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.039172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.039177 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.039191 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.039195 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.039247 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.039346 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.039371 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:23:20.039512 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44479
--builtin_ntp_servers=127.30.160.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:23:20.039687 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.034484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44479 (local address 127.30.160.1:60179)
0808 02:23:20.034707 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:20.034740 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:20.035230 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:20.035445 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:20.035448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:20.035456 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:20.036044 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:20.036051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.036831 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.036832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:20.037696 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:20.037699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.038259 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.038260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:20.038461 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:20.039026 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:23:20.039026 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:20.039065 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:23:20.039092 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:20.039257 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:20.039262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:20.039266 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:23:20.039348 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:23:20.039452 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260808 02:23:20.040488 8640 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44479
I20260808 02:23:20.040591 8640 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:20.040768 8640 heartbeater.cc:511] Master 127.30.160.62:44479 requested a full tablet report, sending...
I20260808 02:23:20.041285 8442 ts_manager.cc:194] Registered new tserver with Master: 0ce2a63c0a4845f0ab589a0605234094 (127.30.160.1:42821)
I20260808 02:23:20.041910 8442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.1:60179
W20260808 02:23:20.125073 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:20.125283 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:20.125305 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:20.126483 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:20.126518 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:20.126543 8643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:23:20.128033 8643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:20.128240 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:20.128501 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:20.133716 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.133737 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.133953 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.133963 8643 server_base.cc:1034] running on GCE node
I20260808 02:23:20.134191 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.134399 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.135565 8643 hybrid_clock.cc:648] HybridClock initialized: now 1786155800135551 us; error 25 us; skew 500 ppm
I20260808 02:23:20.136926 8643 webserver.cc:460] Webserver started at http://127.30.160.2:34541/ using document root <none> and password file <none>
I20260808 02:23:20.137184 8643 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.137257 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.137411 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.138479 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "868c080bb3c049c58bc3d272b572f469"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "55ae3acefb51072f617700285cded62b"
server_key_iv: "593d2cda1b958a836a0c87151c2c9a65"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.138831 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "868c080bb3c049c58bc3d272b572f469"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "55ae3acefb51072f617700285cded62b"
server_key_iv: "593d2cda1b958a836a0c87151c2c9a65"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.140148 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:23:20.140873 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.141042 8643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:20.141141 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "868c080bb3c049c58bc3d272b572f469"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "55ae3acefb51072f617700285cded62b"
server_key_iv: "593d2cda1b958a836a0c87151c2c9a65"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.141218 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.164269 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.164565 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.164671 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.164855 8643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:20.165205 8643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:20.165242 8643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.165297 8643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:20.165336 8643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.171092 8643 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:35289
I20260808 02:23:20.171144 8777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:35289 every 8 connection(s)
I20260808 02:23:20.171478 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:23:20.175423 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8643
I20260808 02:23:20.175524 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:23:20.175736 31360 external_mini_cluster.cc:1428] Setting key 7f8410e4d17b2d054b5d2a0276f4fc01
I20260808 02:23:20.176680 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44479
--builtin_ntp_servers=127.30.160.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:23:20.177211 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.172236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:53173 (local address 127.30.160.62:44479)
0808 02:23:20.172282 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.172285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.172612 (+ 327us) server_negotiation.cc:407] Connection header received
0808 02:23:20.173097 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.173098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.173108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.173157 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.173912 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.174451 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.175575 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.175875 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.175968 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:20.176886 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.176888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.176889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.176894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.176918 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.176922 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.176945 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.177068 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.177097 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:23:20.177317 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.172220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44479 (local address 127.30.160.2:53173)
0808 02:23:20.172450 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:20.172494 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:20.173023 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:20.173218 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:20.173222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:20.173230 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:20.173836 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:20.173842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.174535 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.174536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:20.175438 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:20.175441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.175983 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.175985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:20.176137 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:20.176708 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:23:20.176708 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:20.176738 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:23:20.176760 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:20.176979 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:20.176985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:20.176987 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:23:20.177037 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:23:20.177118 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260808 02:23:20.178670 8778 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44479
I20260808 02:23:20.178753 8778 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:20.178952 8778 heartbeater.cc:511] Master 127.30.160.62:44479 requested a full tablet report, sending...
I20260808 02:23:20.179352 8442 ts_manager.cc:194] Registered new tserver with Master: 868c080bb3c049c58bc3d272b572f469 (127.30.160.2:35289)
I20260808 02:23:20.179728 8442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.2:53173
W20260808 02:23:20.260929 8781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:20.261152 8781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:20.261188 8781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:20.262342 8781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:20.262404 8781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:20.262449 8781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:23:20.263773 8781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:20.264053 8781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:20.264354 8781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:20.269282 8789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.269418 8781 server_base.cc:1034] running on GCE node
W20260808 02:23:20.269294 8786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.269259 8787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.269727 8781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.269927 8781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.271067 8781 hybrid_clock.cc:648] HybridClock initialized: now 1786155800271047 us; error 35 us; skew 500 ppm
I20260808 02:23:20.272547 8781 webserver.cc:460] Webserver started at http://127.30.160.3:37841/ using document root <none> and password file <none>
I20260808 02:23:20.272845 8781 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.272922 8781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.273051 8781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.274173 8781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "ef18e20e075340aea818be56183bbc08"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "7851f17e4ad69f38862bfd9ac1a5c4f0"
server_key_iv: "05c1676aea6df34c5c2999f8adef3adf"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.274488 8781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "ef18e20e075340aea818be56183bbc08"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "7851f17e4ad69f38862bfd9ac1a5c4f0"
server_key_iv: "05c1676aea6df34c5c2999f8adef3adf"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.275790 8781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:23:20.276607 8795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.276787 8781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:20.276890 8781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "ef18e20e075340aea818be56183bbc08"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "7851f17e4ad69f38862bfd9ac1a5c4f0"
server_key_iv: "05c1676aea6df34c5c2999f8adef3adf"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.276983 8781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.298269 8781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.298584 8781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.298743 8781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.298965 8781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:20.299294 8781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:20.299358 8781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.299399 8781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:20.299427 8781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.305339 8781 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:38659
I20260808 02:23:20.305384 8918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:38659 every 8 connection(s)
I20260808 02:23:20.305739 8781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:23:20.311076 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.306506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:54145 (local address 127.30.160.62:44479)
0808 02:23:20.306593 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.306596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.306813 (+ 217us) server_negotiation.cc:407] Connection header received
0808 02:23:20.307371 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.307373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.307385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.307434 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.308231 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.308744 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.309733 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.309954 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.310036 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:20.310797 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.310798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.310799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.310803 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.310820 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.310824 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.310875 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.310967 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.310996 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:23:20.311302 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.306482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44479 (local address 127.30.160.3:54145)
0808 02:23:20.306686 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:20.306719 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:20.307244 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:20.307519 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:20.307523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:20.307531 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:20.308128 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:20.308133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.308856 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.308857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:20.309634 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:20.309636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:20.310046 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:20.310047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:20.310177 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:20.310655 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:23:20.310655 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:20.310683 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:23:20.310706 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:20.310875 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:20.310880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:20.310883 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:20.310968 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:23:20.311069 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260808 02:23:20.311805 8919 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44479
I20260808 02:23:20.311885 8919 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:20.312119 8919 heartbeater.cc:511] Master 127.30.160.62:44479 requested a full tablet report, sending...
I20260808 02:23:20.312533 8442 ts_manager.cc:194] Registered new tserver with Master: ef18e20e075340aea818be56183bbc08 (127.30.160.3:38659)
I20260808 02:23:20.312544 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8781
I20260808 02:23:20.312649 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:23:20.312862 31360 external_mini_cluster.cc:1428] Setting key 527bdb5460fcb512ac01d7b0eb8feeda
I20260808 02:23:20.312985 8442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.3:54145
I20260808 02:23:20.313762 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:23:20.315565 8482 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:20.315058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.30.160.62:44479)
0808 02:23:20.315094 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.315096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.315229 (+ 133us) server_negotiation.cc:407] Connection header received
0808 02:23:20.315292 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.315293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.315303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.315353 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.315491 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60772: BlockingRecv error: recv got EOF from 127.0.0.1:60772 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:23:20.315658 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8504
I20260808 02:23:20.322014 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8643
I20260808 02:23:20.327879 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8781
I20260808 02:23:20.335764 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8409
2026-08-08T02:23:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (622 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T02:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:23:20Z Disabled control of system clock
I20260808 02:23:20.364183 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38677
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:23:20.445729 8931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:20.445919 8931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:20.445940 8931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:20.447045 8931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:23:20.447086 8931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:20.447099 8931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:20.447113 8931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:23:20.447163 8931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:23:20.448578 8931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:38677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:20.448840 8931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:20.449141 8931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:20.453864 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.453878 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.453972 8931 server_base.cc:1034] running on GCE node
W20260808 02:23:20.453894 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.454347 8931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.454638 8931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.455802 8931 hybrid_clock.cc:648] HybridClock initialized: now 1786155800455795 us; error 36 us; skew 500 ppm
I20260808 02:23:20.457043 8931 webserver.cc:460] Webserver started at http://127.30.160.62:32941/ using document root <none> and password file <none>
I20260808 02:23:20.457281 8931 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.457363 8931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.457522 8931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.458627 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "8c185da331e149af9e590a14caba8e01"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "819c46cf3b6aa6d630aae9d974cc6d88"
server_key_iv: "0896cd63af52a7a3c8d7d57c32158d11"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.458976 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "8c185da331e149af9e590a14caba8e01"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "819c46cf3b6aa6d630aae9d974cc6d88"
server_key_iv: "0896cd63af52a7a3c8d7d57c32158d11"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.460287 8931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:23:20.460996 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.461166 8931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.461272 8931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "8c185da331e149af9e590a14caba8e01"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "819c46cf3b6aa6d630aae9d974cc6d88"
server_key_iv: "0896cd63af52a7a3c8d7d57c32158d11"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.461351 8931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.464668 8931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.464928 8931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.465072 8931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.468922 8931 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:38677
I20260808 02:23:20.468953 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:38677 every 8 connection(s)
I20260808 02:23:20.469269 8931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:23:20.469907 9000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:20.472643 9000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01: Bootstrap starting.
I20260808 02:23:20.473215 9000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:20.473516 9000 log.cc:824] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:20.474172 9000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01: No bootstrap required, opened a new log
I20260808 02:23:20.475832 9000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c185da331e149af9e590a14caba8e01" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38677 } }
I20260808 02:23:20.476048 9000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:20.476095 9000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c185da331e149af9e590a14caba8e01, State: Initialized, Role: FOLLOWER
I20260808 02:23:20.476241 9000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c185da331e149af9e590a14caba8e01" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38677 } }
I20260808 02:23:20.476321 9000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:23:20.476375 9000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:23:20.476423 9000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:20.476956 9000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c185da331e149af9e590a14caba8e01" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38677 } }
I20260808 02:23:20.477069 9000 leader_election.cc:302] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c185da331e149af9e590a14caba8e01; no voters:
I20260808 02:23:20.477259 9000 leader_election.cc:288] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:23:20.477319 9004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:20.477489 9004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [term 1 LEADER]: Becoming Leader. State: Replica: 8c185da331e149af9e590a14caba8e01, State: Running, Role: LEADER
I20260808 02:23:20.477517 9000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:23:20.477573 9004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c185da331e149af9e590a14caba8e01" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38677 } }
I20260808 02:23:20.477937 9006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c185da331e149af9e590a14caba8e01. Latest consensus state: current_term: 1 leader_uuid: "8c185da331e149af9e590a14caba8e01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c185da331e149af9e590a14caba8e01" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38677 } } }
I20260808 02:23:20.478029 9006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:20.478152 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c185da331e149af9e590a14caba8e01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c185da331e149af9e590a14caba8e01" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38677 } } }
I20260808 02:23:20.478225 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01 [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:20.478291 9013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:23:20.478542 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 8931
I20260808 02:23:20.478653 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:23:20.478777 9013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:23:20.478870 31360 external_mini_cluster.cc:1428] Setting key abb66ce511408cfc1a80c3f35ee647a2
I20260808 02:23:20.480468 9013 catalog_manager.cc:1347] Generated new cluster ID: 0da604b797a7494fb62a7dc179c3de44
I20260808 02:23:20.480522 9013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:23:20.484618 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.479358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.30.160.62:38677)
0808 02:23:20.479980 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.479989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.480023 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:23:20.480134 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.480142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.480185 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.480335 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.484207 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.484210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.484210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.484213 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.484239 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.484248 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.484333 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.484415 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.484475 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":72,"threads_started":1}
I20260808 02:23:20.485622 9013 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:23:20.485770 9013 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:23:20.499708 9013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c185da331e149af9e590a14caba8e01: Generated new TSK 0
I20260808 02:23:20.499882 9013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:23:20.504995 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38677
--builtin_ntp_servers=127.30.160.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:23:20.588315 9027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:20.588536 9027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:20.588569 9027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:20.589713 9027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:20.589776 9027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:20.589828 9027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:23:20.591192 9027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:20.591459 9027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:20.591756 9027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:20.597456 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.597621 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.597425 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.597929 9027 server_base.cc:1034] running on GCE node
I20260808 02:23:20.598130 9027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.598381 9027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.599543 9027 hybrid_clock.cc:648] HybridClock initialized: now 1786155800599534 us; error 37 us; skew 500 ppm
I20260808 02:23:20.600878 9027 webserver.cc:460] Webserver started at http://127.30.160.1:40811/ using document root <none> and password file <none>
I20260808 02:23:20.601106 9027 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.601174 9027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.601351 9027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.602419 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "590e95ba9b0849d780454149a3550302"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "abad8c4de66b6558afa3aca5bfcea853"
server_key_iv: "a17d1ed366b23f8e98b9da4362664037"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.602788 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "590e95ba9b0849d780454149a3550302"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "abad8c4de66b6558afa3aca5bfcea853"
server_key_iv: "a17d1ed366b23f8e98b9da4362664037"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.604179 9027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:23:20.604977 9041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.605181 9027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:20.605288 9027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "590e95ba9b0849d780454149a3550302"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "abad8c4de66b6558afa3aca5bfcea853"
server_key_iv: "a17d1ed366b23f8e98b9da4362664037"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.605372 9027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.608685 9027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.608948 9027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.609089 9027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.609295 9027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:20.609604 9027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:20.609659 9027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.609704 9027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:20.609732 9027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.615401 9027 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:36269
I20260808 02:23:20.615439 9160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:36269 every 8 connection(s)
I20260808 02:23:20.615784 9027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:23:20.618561 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.616530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:49029 (local address 127.30.160.62:38677)
0808 02:23:20.616610 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.616612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.616809 (+ 197us) server_negotiation.cc:407] Connection header received
0808 02:23:20.617240 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.617241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.617247 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.617300 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.618365 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.618366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.618367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.618370 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.618385 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.618389 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.618437 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.618495 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.618508 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:23:20.618726 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.616493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38677 (local address 127.30.160.1:49029)
0808 02:23:20.616690 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:20.616722 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:20.617160 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:20.617358 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:20.617361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:20.617369 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:20.618219 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:23:20.618220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:20.618246 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:23:20.618273 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:20.618435 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:20.618437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:20.618440 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:20.618490 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:23:20.618536 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260808 02:23:20.619280 9161 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38677
I20260808 02:23:20.619388 9161 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:20.619535 9161 heartbeater.cc:511] Master 127.30.160.62:38677 requested a full tablet report, sending...
I20260808 02:23:20.620100 8962 ts_manager.cc:194] Registered new tserver with Master: 590e95ba9b0849d780454149a3550302 (127.30.160.1:36269)
I20260808 02:23:20.620639 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9027
I20260808 02:23:20.620740 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:23:20.620931 31360 external_mini_cluster.cc:1428] Setting key 8187a667cc414f728589868f95e48279
I20260808 02:23:20.621770 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38677
--builtin_ntp_servers=127.30.160.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:23:20.704444 9165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:20.704667 9165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:20.704703 9165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:20.705837 9165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:20.705900 9165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:20.705943 9165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:23:20.707275 9165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:20.707547 9165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:20.707959 9165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:20.712926 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.712980 9165 server_base.cc:1034] running on GCE node
W20260808 02:23:20.712927 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.712960 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.713395 9165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.713642 9165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.714784 9165 hybrid_clock.cc:648] HybridClock initialized: now 1786155800714770 us; error 28 us; skew 500 ppm
I20260808 02:23:20.716101 9165 webserver.cc:460] Webserver started at http://127.30.160.2:45735/ using document root <none> and password file <none>
I20260808 02:23:20.716359 9165 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.716405 9165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.716573 9165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.717608 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "9d3530c6b7c349f79e9df711a330668b"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "5dc4567f0ebfe153f4d88833c06ba5be"
server_key_iv: "c563150122f304c60a0d71716a0e13d3"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.717967 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "9d3530c6b7c349f79e9df711a330668b"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "5dc4567f0ebfe153f4d88833c06ba5be"
server_key_iv: "c563150122f304c60a0d71716a0e13d3"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.719252 9165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:23:20.720026 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.720218 9165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:20.720290 9165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "9d3530c6b7c349f79e9df711a330668b"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "5dc4567f0ebfe153f4d88833c06ba5be"
server_key_iv: "c563150122f304c60a0d71716a0e13d3"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.720392 9165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.723598 9165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.723855 9165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.724013 9165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.724218 9165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:20.724550 9165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:20.724584 9165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.724638 9165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:20.724666 9165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.730707 9165 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:40721
I20260808 02:23:20.730743 9299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:40721 every 8 connection(s)
I20260808 02:23:20.731086 9165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:23:20.734015 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.731836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:37055 (local address 127.30.160.62:38677)
0808 02:23:20.731947 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.731950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.732143 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:23:20.732579 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.732580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.732586 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.732636 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.733838 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.733839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.733839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.733842 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.733855 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.733858 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.733901 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.733948 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.733959 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:23:20.734211 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.731802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38677 (local address 127.30.160.2:37055)
0808 02:23:20.732021 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:20.732054 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:20.732497 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:20.732702 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:20.732705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:20.732713 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:20.733683 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:23:20.733683 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:20.733727 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:23:20.733756 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:20.733901 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:20.733903 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:20.733906 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:20.733956 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:23:20.734003 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
I20260808 02:23:20.734689 9300 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38677
I20260808 02:23:20.734776 9300 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:20.734892 9300 heartbeater.cc:511] Master 127.30.160.62:38677 requested a full tablet report, sending...
I20260808 02:23:20.735241 8962 ts_manager.cc:194] Registered new tserver with Master: 9d3530c6b7c349f79e9df711a330668b (127.30.160.2:40721)
I20260808 02:23:20.737133 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9165
I20260808 02:23:20.737236 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:23:20.737469 31360 external_mini_cluster.cc:1428] Setting key 77ee7c552495cb79def2a219ea418f94
I20260808 02:23:20.738349 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38677
--builtin_ntp_servers=127.30.160.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:23:20.821287 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:20.821506 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:20.821529 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:20.822741 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:20.822779 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:20.822832 9303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:23:20.824249 9303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:20.824498 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:20.824800 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:23:20.829725 9303 server_base.cc:1034] running on GCE node
W20260808 02:23:20.829694 9310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.829672 9312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:20.829676 9309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:20.830120 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:20.830418 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:20.831583 9303 hybrid_clock.cc:648] HybridClock initialized: now 1786155800831571 us; error 46 us; skew 500 ppm
I20260808 02:23:20.832913 9303 webserver.cc:460] Webserver started at http://127.30.160.3:45991/ using document root <none> and password file <none>
I20260808 02:23:20.833174 9303 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:20.833245 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:20.833401 9303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:20.834468 9303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "1b9ac925b0214197933bdd3f568f92f0"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "a037f86a7a376dfc25d0d4f6ab4312f6"
server_key_iv: "e2599f82bf129a5afe15f25101c0062e"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.834830 9303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "1b9ac925b0214197933bdd3f568f92f0"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "a037f86a7a376dfc25d0d4f6ab4312f6"
server_key_iv: "e2599f82bf129a5afe15f25101c0062e"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.836182 9303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:23:20.836916 9318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.837090 9303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:20.837198 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "1b9ac925b0214197933bdd3f568f92f0"
format_stamp: "Formatted at 2026-08-08 02:23:20 on dist-test-slave-zkpd"
server_key: "a037f86a7a376dfc25d0d4f6ab4312f6"
server_key_iv: "e2599f82bf129a5afe15f25101c0062e"
server_key_version: "encryptionkey@0"
I20260808 02:23:20.837273 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:20.840706 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:20.840973 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:20.841114 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:20.841324 9303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:20.841629 9303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:20.841688 9303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.841727 9303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:20.841755 9303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:20.847589 9303 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:46711
I20260808 02:23:20.847625 9438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:46711 every 8 connection(s)
I20260808 02:23:20.848021 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:23:20.850934 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.848733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:60635 (local address 127.30.160.62:38677)
0808 02:23:20.848794 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.848796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.849025 (+ 229us) server_negotiation.cc:407] Connection header received
0808 02:23:20.849533 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.849534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.849542 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.849594 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.850678 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:20.850679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:20.850680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:23:20.850684 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:20.850700 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:23:20.850705 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:20.850765 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:20.850850 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:23:20.850866 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:23:20.851128 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:20.848736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38677 (local address 127.30.160.3:60635)
0808 02:23:20.848911 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:20.848943 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:20.849415 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:20.849660 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:20.849664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:20.849674 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:20.850531 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:23:20.850532 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:20.850557 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:23:20.850583 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:20.850765 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:20.850771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:20.850773 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:23:20.850858 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:23:20.850916 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260808 02:23:20.851639 9439 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38677
I20260808 02:23:20.851727 9439 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:20.851853 9439 heartbeater.cc:511] Master 127.30.160.62:38677 requested a full tablet report, sending...
I20260808 02:23:20.852252 8962 ts_manager.cc:194] Registered new tserver with Master: 1b9ac925b0214197933bdd3f568f92f0 (127.30.160.3:46711)
I20260808 02:23:20.853894 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9303
I20260808 02:23:20.853974 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:23:20.854120 31360 external_mini_cluster.cc:1428] Setting key 8a1dd240501d47d60ffafedc816938dc
I20260808 02:23:20.854921 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:23:20.856621 9022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:23:20.856191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.30.160.62:38677)
0808 02:23:20.856262 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:20.856264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:20.856364 (+ 100us) server_negotiation.cc:407] Connection header received
0808 02:23:20.856400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:20.856401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:20.856404 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:20.856448 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:20.856567 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55556: BlockingRecv error: recv got EOF from 127.0.0.1:55556 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:23:20.856715 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9027
I20260808 02:23:20.864218 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9165
I20260808 02:23:20.871589 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9303
I20260808 02:23:20.876901 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 8931
2026-08-08T02:23:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (540 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:23:20 dist-test-slave-zkpd krb5kdc[9450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:23:20 dist-test-slave-zkpd krb5kdc[9450](info): set up 2 sockets
Aug 08 02:23:20 dist-test-slave-zkpd krb5kdc[9450](info): commencing operation
krb5kdc: starting...
W20260808 02:23:21.941000 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:23:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:23:22.063402 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37869
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:22.146999 9463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:22.147241 9463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:22.147275 9463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:22.148440 9463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:23:22.148516 9463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:22.148540 9463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:22.148562 9463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:23:22.148591 9463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:23:22.150070 9463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:22.150352 9463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:22.150581 9463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:22.155503 9470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:22.155476 9469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:22.155539 9463 server_base.cc:1034] running on GCE node
W20260808 02:23:22.155539 9472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:22.155947 9463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:22.156257 9463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:22.157402 9463 hybrid_clock.cc:648] HybridClock initialized: now 1786155802157391 us; error 32 us; skew 500 ppm
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:22.160029 9463 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:23:22.160458 9463 webserver.cc:460] Webserver started at http://127.30.160.62:44725/ using document root <none> and password file <none>
I20260808 02:23:22.160655 9463 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:22.160696 9463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:22.160791 9463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:22.161783 9463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "2a7a491497b9494fb28f6d67129fee3b"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "16d2f28e75a80b7d0a275773fd9c6d32"
server_key_iv: "3a37a5173c1508ac7fb1041f860016fd"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.162099 9463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "2a7a491497b9494fb28f6d67129fee3b"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "16d2f28e75a80b7d0a275773fd9c6d32"
server_key_iv: "3a37a5173c1508ac7fb1041f860016fd"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.163347 9463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:23:22.164047 9479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.164223 9463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:22.164318 9463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "2a7a491497b9494fb28f6d67129fee3b"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "16d2f28e75a80b7d0a275773fd9c6d32"
server_key_iv: "3a37a5173c1508ac7fb1041f860016fd"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.164420 9463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:22.192101 9463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:22.194615 9463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:22.194793 9463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:22.199112 9463 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:37869
I20260808 02:23:22.199155 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:37869 every 8 connection(s)
I20260808 02:23:22.199554 9463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:23:22.200646 9535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:22.202981 9535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b: Bootstrap starting.
I20260808 02:23:22.203591 9535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:22.203914 9535 log.cc:824] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:22.204643 9535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b: No bootstrap required, opened a new log
I20260808 02:23:22.206254 9535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7a491497b9494fb28f6d67129fee3b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37869 } }
I20260808 02:23:22.206399 9535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:22.206422 9535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7a491497b9494fb28f6d67129fee3b, State: Initialized, Role: FOLLOWER
I20260808 02:23:22.206575 9535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7a491497b9494fb28f6d67129fee3b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37869 } }
I20260808 02:23:22.206651 9535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:23:22.206676 9535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:23:22.206739 9535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:22.207567 9535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7a491497b9494fb28f6d67129fee3b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37869 } }
I20260808 02:23:22.207659 9535 leader_election.cc:302] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a7a491497b9494fb28f6d67129fee3b; no voters:
I20260808 02:23:22.207839 9535 leader_election.cc:288] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:23:22.207940 9538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:22.208096 9538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [term 1 LEADER]: Becoming Leader. State: Replica: 2a7a491497b9494fb28f6d67129fee3b, State: Running, Role: LEADER
I20260808 02:23:22.208127 9535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:23:22.208213 9538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7a491497b9494fb28f6d67129fee3b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37869 } }
I20260808 02:23:22.208617 9539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a7a491497b9494fb28f6d67129fee3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7a491497b9494fb28f6d67129fee3b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37869 } } }
I20260808 02:23:22.208678 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9463
I20260808 02:23:22.208789 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:23:22.208868 9539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:22.209022 31360 external_mini_cluster.cc:1428] Setting key 3cf8d8a45f822157200d7d59d7b64718
I20260808 02:23:22.208786 9540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7a491497b9494fb28f6d67129fee3b. Latest consensus state: current_term: 1 leader_uuid: "2a7a491497b9494fb28f6d67129fee3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7a491497b9494fb28f6d67129fee3b" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37869 } } }
I20260808 02:23:22.209219 9540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b [sys.catalog]: This master's current role is: LEADER
I20260808 02:23:22.209369 9547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:23:22.209868 9547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:23:22.211450 9547 catalog_manager.cc:1347] Generated new cluster ID: dcfcff2e020e45c6ae79744339e72293
I20260808 02:23:22.211506 9547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:22.220475 9547 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:23:22.221155 9547 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:23:22.221807 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.209712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.30.160.62:37869)
0808 02:23:22.210091 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.210102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.210139 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:23:22.210480 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.210489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.210552 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.210701 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:22.211546 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.212979 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.214028 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.214676 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.214794 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.217694 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:22.217698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:22.217698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:22.217711 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:22.219841 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.220466 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.220469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.220469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.220605 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.220847 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.220848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.220848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.221151 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:22.221323 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.221432 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.221570 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260808 02:23:22.229862 9547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b: Generated new TSK 0
I20260808 02:23:22.230057 9547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:23:22.260759 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37869
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:22.346930 9562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:22.347129 9562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:22.347152 9562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:22.348315 9562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:22.348357 9562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:22.348383 9562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:23:22.349870 9562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:22.350121 9562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:22.350369 9562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:22.355453 9567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:22.355450 9568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:22.355686 9562 server_base.cc:1034] running on GCE node
W20260808 02:23:22.355477 9570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:22.356019 9562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:22.356279 9562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:22.357436 9562 hybrid_clock.cc:648] HybridClock initialized: now 1786155802357430 us; error 41 us; skew 500 ppm
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:22.360117 9562 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:23:22.360483 9562 webserver.cc:460] Webserver started at http://127.30.160.1:44695/ using document root <none> and password file <none>
I20260808 02:23:22.360680 9562 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:22.360755 9562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:22.360859 9562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:22.361804 9562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "19fa69a977294aaa9e0544328ad69480"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "4450ea5aca1755b668d0c2754fc72b92"
server_key_iv: "d727b9958bb6741c41b83f61824297fa"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.362102 9562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "19fa69a977294aaa9e0544328ad69480"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "4450ea5aca1755b668d0c2754fc72b92"
server_key_iv: "d727b9958bb6741c41b83f61824297fa"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.363328 9562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:23:22.364044 9578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.364230 9562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:23:22.364331 9562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "19fa69a977294aaa9e0544328ad69480"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "4450ea5aca1755b668d0c2754fc72b92"
server_key_iv: "d727b9958bb6741c41b83f61824297fa"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.364419 9562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:22.394378 9562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:22.396415 9562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:22.396577 9562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:22.396809 9562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:22.397143 9562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:22.397197 9562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.397239 9562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:22.397276 9562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.403172 9562 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:35415
I20260808 02:23:22.403223 9700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:35415 every 8 connection(s)
I20260808 02:23:22.403530 9562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:23:22.406258 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9562
I20260808 02:23:22.406343 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:23:22.406545 31360 external_mini_cluster.cc:1428] Setting key 6e7ac070e03d7f9c42fae85f65ed01b8
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:22.414073 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.404307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:48481 (local address 127.30.160.62:37869)
0808 02:23:22.404351 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.404353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.404594 (+ 241us) server_negotiation.cc:407] Connection header received
0808 02:23:22.405138 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.405139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.405152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.405206 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:22.406175 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.406924 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.408462 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.408709 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.408785 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.411080 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:22.411082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:22.411082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:22.411090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:22.412618 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.413058 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.413059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.413060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.413120 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.413382 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.413383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.413384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.413500 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:22.413573 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.413886 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.413981 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:23:22.414237 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.404275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37869 (local address 127.30.160.1:48481)
0808 02:23:22.404475 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.404510 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.405026 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.405279 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.405283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.405878 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:22.406069 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.406074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.407000 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.407001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.408377 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.408381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.408859 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.408861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.408971 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.409578 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:22.409587 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:22.410980 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:22.412724 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:22.412732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:22.412734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:22.412968 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:22.413195 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:22.413196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:22.413196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:22.413307 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:22.413576 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:22.413584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:22.413732 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.413842 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.414006 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260808 02:23:22.414906 9701 heartbeater.cc:347] Connected to a master server at 127.30.160.62:37869
I20260808 02:23:22.415115 9701 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:22.415305 9701 heartbeater.cc:511] Master 127.30.160.62:37869 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
I20260808 02:23:22.415959 9499 ts_manager.cc:194] Registered new tserver with Master: 19fa69a977294aaa9e0544328ad69480 (127.30.160.1:35415)
I20260808 02:23:22.416733 9499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:48481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:23:22.432801 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37869
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:22.515364 9706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:22.515560 9706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:22.515583 9706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:22.516790 9706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:22.516842 9706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:22.516868 9706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:23:22.518332 9706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:22.518581 9706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:22.518823 9706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:22.523679 9714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:22.523696 9711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:22.523718 9712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:22.524004 9706 server_base.cc:1034] running on GCE node
I20260808 02:23:22.524210 9706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:22.524482 9706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:22.525676 9706 hybrid_clock.cc:648] HybridClock initialized: now 1786155802525656 us; error 30 us; skew 500 ppm
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:22.528360 9706 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:23:22.528853 9706 webserver.cc:460] Webserver started at http://127.30.160.2:38151/ using document root <none> and password file <none>
I20260808 02:23:22.529109 9706 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:22.529182 9706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:22.529345 9706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:22.530383 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "44c71719b8a342e2a0aee242e80be57a"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "31feed3e122fab32db4caab3e4cf7e87"
server_key_iv: "310a6ea19555d70de89d4dd606c494d9"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.530788 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "44c71719b8a342e2a0aee242e80be57a"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "31feed3e122fab32db4caab3e4cf7e87"
server_key_iv: "310a6ea19555d70de89d4dd606c494d9"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.532136 9706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:23:22.532866 9721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.533047 9706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:22.533162 9706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "44c71719b8a342e2a0aee242e80be57a"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "31feed3e122fab32db4caab3e4cf7e87"
server_key_iv: "310a6ea19555d70de89d4dd606c494d9"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.533241 9706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:22.552482 9706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:22.554477 9706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:22.554661 9706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:22.554911 9706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:22.555230 9706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:22.555289 9706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.555331 9706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:22.555361 9706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.561645 9706 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:41621
I20260808 02:23:22.561700 9843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:41621 every 8 connection(s)
I20260808 02:23:22.562013 9706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:23:22.567734 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9706
I20260808 02:23:22.567812 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:23:22.568025 31360 external_mini_cluster.cc:1428] Setting key 1bd4c71438058118f1668099cee554ad
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:22.572120 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.562706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:50221 (local address 127.30.160.62:37869)
0808 02:23:22.562782 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.562785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.562991 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:23:22.563481 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.563481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.563492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.563546 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:22.564528 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.565028 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.565993 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.566182 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.566239 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.568928 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:22.568930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:22.568930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:22.568939 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:22.570568 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.571105 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.571106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.571107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.571170 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.571468 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.571469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.571469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.571573 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:22.571626 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.571981 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.572057 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:23:22.572418 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.562680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37869 (local address 127.30.160.2:50221)
0808 02:23:22.562861 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.562903 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.563396 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.563607 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.563610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.564233 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:22.564443 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.564448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.565110 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.565110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.565912 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.565914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.566258 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.566259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.566371 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.566962 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:22.566971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:22.568831 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:22.570667 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:22.570676 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:22.570678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:22.571009 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:22.571254 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:22.571255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:22.571256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:22.571401 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:22.571638 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:22.571644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:22.571804 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.571950 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.572129 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260808 02:23:22.572952 9844 heartbeater.cc:347] Connected to a master server at 127.30.160.62:37869
I20260808 02:23:22.573050 9844 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:22.573247 9844 heartbeater.cc:511] Master 127.30.160.62:37869 requested a full tablet report, sending...
I20260808 02:23:22.573616 9499 ts_manager.cc:194] Registered new tserver with Master: 44c71719b8a342e2a0aee242e80be57a (127.30.160.2:41621)
I20260808 02:23:22.574005 9499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:50221
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:23:22.595340 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37869
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:23:22.678411 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:23:22.678594 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:23:22.678617 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:23:22.679709 9851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:23:22.679746 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:23:22.679796 9851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:23:22.681282 9851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:23:22.681463 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:23:22.681639 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:23:22.686451 9856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:22.686491 9857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:23:22.686498 9859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:23:22.686630 9851 server_base.cc:1034] running on GCE node
I20260808 02:23:22.686884 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:23:22.687105 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:23:22.688259 9851 hybrid_clock.cc:648] HybridClock initialized: now 1786155802688234 us; error 39 us; skew 500 ppm
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:23:22.690795 9851 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:23:22.691244 9851 webserver.cc:460] Webserver started at http://127.30.160.3:44317/ using document root <none> and password file <none>
I20260808 02:23:22.691480 9851 fs_manager.cc:359] Metadata directory not provided
I20260808 02:23:22.691525 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:23:22.691677 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:23:22.692749 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "a423079cd43a48c8b58068a2d0de8b0f"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "acb0d928ad865aabefea86094db9adc7"
server_key_iv: "21a9402a044d796c21b81e8df398e66b"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.693130 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "a423079cd43a48c8b58068a2d0de8b0f"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "acb0d928ad865aabefea86094db9adc7"
server_key_iv: "21a9402a044d796c21b81e8df398e66b"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.694480 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:23:22.695207 9866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.695402 9851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:23:22.695508 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "a423079cd43a48c8b58068a2d0de8b0f"
format_stamp: "Formatted at 2026-08-08 02:23:22 on dist-test-slave-zkpd"
server_key: "acb0d928ad865aabefea86094db9adc7"
server_key_iv: "21a9402a044d796c21b81e8df398e66b"
server_key_version: "encryptionkey@0"
I20260808 02:23:22.695577 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:23:22.719816 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:23:22.722342 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:23:22.722539 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:23:22.722780 9851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:23:22.723119 9851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:23:22.723174 9851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.723209 9851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:23:22.723237 9851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:23:22.729295 9851 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:46621
I20260808 02:23:22.729339 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:46621 every 8 connection(s)
I20260808 02:23:22.729660 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:23:22.730563 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 9851
I20260808 02:23:22.730682 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:23:22.730890 31360 external_mini_cluster.cc:1428] Setting key 869af30287ac7081c5c0ac23679387ed
Aug 08 02:23:22 dist-test-slave-zkpd krb5kdc[9450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:23:22.738924 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.730369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:60987 (local address 127.30.160.62:37869)
0808 02:23:22.730447 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.730450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.730695 (+ 245us) server_negotiation.cc:407] Connection header received
0808 02:23:22.731348 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.731349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.731363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.731424 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:22.732484 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.732990 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.733950 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.734134 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.734186 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.736261 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:22.736262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:22.736262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:22.736269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:22.737831 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.738169 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.738169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.738170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.738219 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.738428 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.738428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.738428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.738527 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:22.738576 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.738783 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.738859 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:23:22.739131 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.730364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:37869 (local address 127.30.160.3:60987)
0808 02:23:22.730545 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.730582 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.731237 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.731524 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.731529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.732197 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:23:22.732405 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.732410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.733069 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.733070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.733874 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.733877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.734207 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.734207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.734316 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.734768 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:23:22.734777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:23:22.736180 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:23:22.737897 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:22.737903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:22.737905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:22.738105 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:22.738275 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:23:22.738275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:23:22.738275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:23:22.738374 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:23:22.738580 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:23:22.738585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:23:22.738693 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.738772 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.738886 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260808 02:23:22.739593 9987 heartbeater.cc:347] Connected to a master server at 127.30.160.62:37869
I20260808 02:23:22.739672 9987 heartbeater.cc:464] Registering TS with master...
I20260808 02:23:22.739861 9987 heartbeater.cc:511] Master 127.30.160.62:37869 requested a full tablet report, sending...
I20260808 02:23:22.740272 9499 ts_manager.cc:194] Registered new tserver with Master: a423079cd43a48c8b58068a2d0de8b0f (127.30.160.3:46621)
I20260808 02:23:22.740648 9499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:60987
I20260808 02:23:22.742556 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:23:22.749696 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.743806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.30.160.62:37869)
0808 02:23:22.743862 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.743865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.743950 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:23:22.744027 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.744028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.744038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.744109 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:23:22.744669 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.745128 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.745878 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.746062 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.746136 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.747324 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:23:22.747326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:23:22.747326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:23:22.747333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:23:22.748528 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.748882 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.748883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.748883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.748937 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:23:22.749142 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:23:22.749143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:23:22.749143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:23:22.749285 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:23:22.749394 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.749524 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.749597 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:23:22.751130 9499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:23:22.752761 9499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:23:22.758965 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.754909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:41621 (local address 127.0.0.1:39188)
0808 02:23:22.755260 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.755302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.755383 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.755773 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.755774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.755776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:22.755843 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.755845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.756734 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.756735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.757883 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.757886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.758687 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.758687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.758740 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.758749 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.758841 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.758893 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":65,"threads_started":1}
I20260808 02:23:22.758973 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.755013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.30.160.2:41621)
0808 02:23:22.755190 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.755195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.755395 (+ 200us) server_negotiation.cc:407] Connection header received
0808 02:23:22.755513 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.755517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.755534 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.755590 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:22.755954 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.756642 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.757982 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.758567 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.758602 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.758638 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.758856 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.758893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260808 02:23:22.759496 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.754933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:35415 (local address 127.0.0.1:42426)
0808 02:23:22.755261 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.755302 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.755381 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.755628 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.755631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.755633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:22.755697 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.755699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.757115 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.757116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.758163 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.758166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.759245 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.759246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.759306 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.759310 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.759368 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.759416 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":93,"threads_started":1}
I20260808 02:23:22.759522 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.755078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.30.160.1:35415)
0808 02:23:22.755267 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.755271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.755434 (+ 163us) server_negotiation.cc:407] Connection header received
0808 02:23:22.755483 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.755486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.755504 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.755581 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:22.755785 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.756999 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.758244 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.758944 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.759005 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.759212 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.759401 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.759446 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260808 02:23:22.759744 9771 tablet_service.cc:1467] Processing CreateTablet for tablet 1d33ddf1d22c44349999799bfc9e4846 (DEFAULT_TABLE table=test-table [id=7d93276c302545c7bb7c6aaa6ef0ea33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:22.760041 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.755118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:46621 (local address 127.0.0.1:54990)
0808 02:23:22.755489 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.755499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.755538 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.755852 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.755853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.755855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:22.755924 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.755925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.757141 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.757142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.758028 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.758030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.759006 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.759007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.759364 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.759379 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.759426 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.759457 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260808 02:23:22.760203 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d33ddf1d22c44349999799bfc9e4846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:22.760310 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.755340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.30.160.3:46621)
0808 02:23:22.755519 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.755524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.755535 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:23:22.755675 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.755678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.755695 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.755768 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:22.755986 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.757051 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.758120 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.758836 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.758908 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.758968 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.759542 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.759832 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260808 02:23:22.760375 9631 tablet_service.cc:1467] Processing CreateTablet for tablet 1d33ddf1d22c44349999799bfc9e4846 (DEFAULT_TABLE table=test-table [id=7d93276c302545c7bb7c6aaa6ef0ea33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:22.760708 9631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d33ddf1d22c44349999799bfc9e4846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:22.760908 9918 tablet_service.cc:1467] Processing CreateTablet for tablet 1d33ddf1d22c44349999799bfc9e4846 (DEFAULT_TABLE table=test-table [id=7d93276c302545c7bb7c6aaa6ef0ea33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:23:22.761221 9918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d33ddf1d22c44349999799bfc9e4846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:23:22.763813 10005 tablet_bootstrap.cc:492] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Bootstrap starting.
I20260808 02:23:22.764101 10006 tablet_bootstrap.cc:492] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Bootstrap starting.
I20260808 02:23:22.764279 10007 tablet_bootstrap.cc:492] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: Bootstrap starting.
I20260808 02:23:22.764686 10005 tablet_bootstrap.cc:654] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:22.764760 10006 tablet_bootstrap.cc:654] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:22.764962 10007 tablet_bootstrap.cc:654] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: Neither blocks nor log segments found. Creating new log.
I20260808 02:23:22.765383 10005 log.cc:824] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:22.765388 10006 log.cc:824] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:22.765499 10007 log.cc:824] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: Log is configured to *not* fsync() on all Append() calls
I20260808 02:23:22.766140 10005 tablet_bootstrap.cc:492] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: No bootstrap required, opened a new log
I20260808 02:23:22.766224 10005 ts_tablet_manager.cc:1397] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:23:22.766274 10007 tablet_bootstrap.cc:492] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: No bootstrap required, opened a new log
I20260808 02:23:22.766373 10007 ts_tablet_manager.cc:1397] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:23:22.768201 10007 raft_consensus.cc:348] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.768313 10006 tablet_bootstrap.cc:492] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: No bootstrap required, opened a new log
I20260808 02:23:22.768353 10007 raft_consensus.cc:374] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:22.768380 10006 ts_tablet_manager.cc:1397] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 02:23:22.768409 10007 raft_consensus.cc:729] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fa69a977294aaa9e0544328ad69480, State: Initialized, Role: FOLLOWER
I20260808 02:23:22.768337 10005 raft_consensus.cc:348] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.768514 10005 raft_consensus.cc:374] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:22.768550 10005 raft_consensus.cc:729] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c71719b8a342e2a0aee242e80be57a, State: Initialized, Role: FOLLOWER
I20260808 02:23:22.768537 10007 consensus_queue.cc:260] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.768659 10005 consensus_queue.cc:260] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.768839 10007 ts_tablet_manager.cc:1428] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 02:23:22.768885 9701 heartbeater.cc:503] Master 127.30.160.62:37869 was elected leader, sending a full tablet report...
I20260808 02:23:22.768940 10005 ts_tablet_manager.cc:1428] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:23:22.769078 9844 heartbeater.cc:503] Master 127.30.160.62:37869 was elected leader, sending a full tablet report...
I20260808 02:23:22.770442 10006 raft_consensus.cc:348] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.770555 10006 raft_consensus.cc:374] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:23:22.770576 10006 raft_consensus.cc:729] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a423079cd43a48c8b58068a2d0de8b0f, State: Initialized, Role: FOLLOWER
I20260808 02:23:22.770678 10006 consensus_queue.cc:260] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.770895 10006 ts_tablet_manager.cc:1428] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:23:22.770957 9987 heartbeater.cc:503] Master 127.30.160.62:37869 was elected leader, sending a full tablet report...
W20260808 02:23:22.812565 9845 tablet.cc:2369] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:23:22.897543 10013 raft_consensus.cc:482] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:23:22.897675 10013 raft_consensus.cc:504] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.898404 10013 leader_election.cc:288] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fa69a977294aaa9e0544328ad69480 (127.30.160.1:35415), a423079cd43a48c8b58068a2d0de8b0f (127.30.160.3:46621)
I20260808 02:23:22.901543 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.898426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:44169 (local address 127.30.160.1:35415)
0808 02:23:22.898478 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.898480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.898490 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:22.898569 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.898570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.898583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.898655 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:22.898885 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.899532 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.900756 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.901151 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.901241 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.901251 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.901423 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.901465 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:23:22.901557 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.898803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:48343 (local address 127.30.160.3:46621)
0808 02:23:22.898875 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.898877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.898887 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:23:22.898916 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.898917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.898926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.898985 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:23:22.899149 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.899642 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.900804 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.901185 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.901269 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.901277 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.901449 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.901487 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:23:22.901573 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.898332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:35415 (local address 127.30.160.2:44169)
0808 02:23:22.898402 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.898417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.898483 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.898673 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.898674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.898675 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:22.898759 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.898761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.899617 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.899618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.900663 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.900665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.901251 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.901253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.901349 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.901352 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.901417 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.901461 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":28}
I20260808 02:23:22.901578 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.898599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:46621 (local address 127.30.160.2:48343)
0808 02:23:22.898764 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.898774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:23:22.898816 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:23:22.899006 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:23:22.899007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:23:22.899008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:23:22.899070 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.899071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.899736 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.899737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.900714 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:23:22.900717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.901285 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.901286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:23:22.901373 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.901376 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:23:22.901444 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:23:22.901483 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260808 02:23:22.901993 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d33ddf1d22c44349999799bfc9e4846" candidate_uuid: "44c71719b8a342e2a0aee242e80be57a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a423079cd43a48c8b58068a2d0de8b0f" is_pre_election: true
I20260808 02:23:22.901994 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d33ddf1d22c44349999799bfc9e4846" candidate_uuid: "44c71719b8a342e2a0aee242e80be57a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fa69a977294aaa9e0544328ad69480" is_pre_election: true
I20260808 02:23:22.902158 9651 raft_consensus.cc:2454] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c71719b8a342e2a0aee242e80be57a in term 0.
I20260808 02:23:22.902158 9939 raft_consensus.cc:2454] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c71719b8a342e2a0aee242e80be57a in term 0.
I20260808 02:23:22.902421 9723 leader_election.cc:302] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fa69a977294aaa9e0544328ad69480, 44c71719b8a342e2a0aee242e80be57a; no voters:
I20260808 02:23:22.902563 10013 raft_consensus.cc:2781] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:23:22.902624 10013 raft_consensus.cc:482] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:23:22.902652 10013 raft_consensus.cc:3037] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:22.903345 10013 raft_consensus.cc:504] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.903503 10013 leader_election.cc:288] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [CANDIDATE]: Term 1 election: Requested vote from peers 19fa69a977294aaa9e0544328ad69480 (127.30.160.1:35415), a423079cd43a48c8b58068a2d0de8b0f (127.30.160.3:46621)
I20260808 02:23:22.903671 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d33ddf1d22c44349999799bfc9e4846" candidate_uuid: "44c71719b8a342e2a0aee242e80be57a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fa69a977294aaa9e0544328ad69480"
I20260808 02:23:22.903698 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d33ddf1d22c44349999799bfc9e4846" candidate_uuid: "44c71719b8a342e2a0aee242e80be57a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a423079cd43a48c8b58068a2d0de8b0f"
I20260808 02:23:22.903771 9939 raft_consensus.cc:3037] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:23:22.903782 9651 raft_consensus.cc:3037] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 0 FOLLOWER]: Advancing to term 1
W20260808 02:23:22.904217 9702 tablet.cc:2369] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:23:22.904685 9939 raft_consensus.cc:2454] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c71719b8a342e2a0aee242e80be57a in term 1.
I20260808 02:23:22.904825 9651 raft_consensus.cc:2454] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c71719b8a342e2a0aee242e80be57a in term 1.
I20260808 02:23:22.904942 9722 leader_election.cc:302] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c71719b8a342e2a0aee242e80be57a, a423079cd43a48c8b58068a2d0de8b0f; no voters:
I20260808 02:23:22.905104 10013 raft_consensus.cc:2781] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:23:22.905248 10013 raft_consensus.cc:686] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 1 LEADER]: Becoming Leader. State: Replica: 44c71719b8a342e2a0aee242e80be57a, State: Running, Role: LEADER
I20260808 02:23:22.905366 10013 consensus_queue.cc:237] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } }
I20260808 02:23:22.906292 9498 catalog_manager.cc:5495] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c71719b8a342e2a0aee242e80be57a (127.30.160.2). New cstate: current_term: 1 leader_uuid: "44c71719b8a342e2a0aee242e80be57a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c71719b8a342e2a0aee242e80be57a" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 41621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 } health_report { overall_health: UNKNOWN } } }
I20260808 02:23:22.941056 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.938489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.30.160.2:41621)
0808 02:23:22.938558 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:23:22.938560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:23:22.938571 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:23:22.938646 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:23:22.938647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:23:22.938672 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:23:22.938764 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:23:22.938923 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.939411 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.940220 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:23:22.940437 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:23:22.940514 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:23:22.940686 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:23:22.940800 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:23:22.940903 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:23:22.940923 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 02:23:22.940944 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:23:22.943040 9651 raft_consensus.cc:1261] T 1d33ddf1d22c44349999799bfc9e4846 P 19fa69a977294aaa9e0544328ad69480 [term 1 FOLLOWER]: Refusing update from remote peer 44c71719b8a342e2a0aee242e80be57a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:22.943082 9939 raft_consensus.cc:1261] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 1 FOLLOWER]: Refusing update from remote peer 44c71719b8a342e2a0aee242e80be57a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:23:22.943497 10016 consensus_queue.cc:1035] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fa69a977294aaa9e0544328ad69480" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:22.943616 10013 consensus_queue.cc:1035] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a423079cd43a48c8b58068a2d0de8b0f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:23:22.946030 10019 mvcc.cc:205] Tried to move back new op lower bound from 7316094168852008960 to 7316094168701009920. Current Snapshot: MvccSnapshot[applied={T|T < 7316094168852008960}]
I20260808 02:23:22.946574 10020 mvcc.cc:205] Tried to move back new op lower bound from 7316094168852008960 to 7316094168701009920. Current Snapshot: MvccSnapshot[applied={T|T < 7316094168852008960}]
I20260808 02:23:22.948472 9499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:22.948573 9499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:23:22.949386 9499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a7a491497b9494fb28f6d67129fee3b: Sending DeleteTablet for 3 replicas of tablet 1d33ddf1d22c44349999799bfc9e4846
I20260808 02:23:22.949726 9631 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d33ddf1d22c44349999799bfc9e4846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:22 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42426
I20260808 02:23:22.949801 9918 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d33ddf1d22c44349999799bfc9e4846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:22 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:54990
I20260808 02:23:22.949836 9771 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d33ddf1d22c44349999799bfc9e4846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:23:22 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:39188
I20260808 02:23:22.950093 10030 tablet_replica.cc:330] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: stopping tablet replica
I20260808 02:23:22.950093 10029 tablet_replica.cc:330] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: stopping tablet replica
I20260808 02:23:22.950223 10030 raft_consensus.cc:2229] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:23:22.950222 10029 raft_consensus.cc:2229] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:23:22.950328 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9562
I20260808 02:23:22.950361 10030 raft_consensus.cc:2258] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:22.950366 10029 raft_consensus.cc:2258] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:23:22.951578 10029 ts_tablet_manager.cc:1905] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:22.951884 10030 ts_tablet_manager.cc:1905] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:23:22.953404 10029 ts_tablet_manager.cc:1918] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:22.953476 10029 log.cc:1196] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/1d33ddf1d22c44349999799bfc9e4846
I20260808 02:23:22.953629 10030 ts_tablet_manager.cc:1918] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:23:22.953694 10030 log.cc:1196] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/1d33ddf1d22c44349999799bfc9e4846
I20260808 02:23:22.953794 10029 ts_tablet_manager.cc:1939] T 1d33ddf1d22c44349999799bfc9e4846 P a423079cd43a48c8b58068a2d0de8b0f: Deleting consensus metadata
I20260808 02:23:22.953986 10030 ts_tablet_manager.cc:1939] T 1d33ddf1d22c44349999799bfc9e4846 P 44c71719b8a342e2a0aee242e80be57a: Deleting consensus metadata
I20260808 02:23:22.955087 9480 catalog_manager.cc:4841] TS a423079cd43a48c8b58068a2d0de8b0f (127.30.160.3:46621): tablet 1d33ddf1d22c44349999799bfc9e4846 (table test-table [id=7d93276c302545c7bb7c6aaa6ef0ea33]) successfully deleted
I20260808 02:23:22.955188 9480 catalog_manager.cc:4841] TS 44c71719b8a342e2a0aee242e80be57a (127.30.160.2:41621): tablet 1d33ddf1d22c44349999799bfc9e4846 (table test-table [id=7d93276c302545c7bb7c6aaa6ef0ea33]) successfully deleted
W20260808 02:23:22.960755 9481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:35415 (error 108)
I20260808 02:23:22.961073 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9706
I20260808 02:23:22.961998 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:23:22.961201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:35415 (local address 127.0.0.1:42428)
0808 02:23:22.961912 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0808 02:23:22.961956 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689}
W20260808 02:23:22.962325 9481 catalog_manager.cc:4584] TS 19fa69a977294aaa9e0544328ad69480 (127.30.160.1:35415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d33ddf1d22c44349999799bfc9e4846: Network error: Client connection negotiation failed: client connection to 127.30.160.1:35415: connect: Connection refused (error 111)
I20260808 02:23:22.967732 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9851
I20260808 02:23:22.974536 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 9463
2026-08-08T02:23:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2099 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:23:23 dist-test-slave-zkpd krb5kdc[10034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:23:23 dist-test-slave-zkpd krb5kdc[10034](info): set up 2 sockets
Aug 08 02:23:23 dist-test-slave-zkpd krb5kdc[10034](info): commencing operation
krb5kdc: starting...
W20260808 02:23:24.061545 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:23:24 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:23:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/rangeradmin_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/rangeradmin_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/rangerlookup_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/rangerlookup_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
I20260808 02:23:24.224479 31360 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8DZQyQ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/postgres -l logfile start
2026-08-08 02:23:26.696 UTC [10065] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:23:26.696 UTC [10065] LOG: listening on IPv4 address "127.30.160.20", port 38813
2026-08-08 02:23:26.699 UTC [10065] LOG: listening on Unix socket "/tmp/.s.PGSQL.38813"
2026-08-08 02:23:26.721 UTC [10069] LOG: database system was shut down at 2026-08-08 02:23:25 UTC
2026-08-08 02:23:26.724 UTC [10065] LOG: database system is ready to accept connections
I20260808 02:23:27.748980 31360 mini_postgres.cc:96] Postgres bound to 38813
2026-08-08 02:23:27.756 UTC [10078] FATAL: database "slave" does not exist
127.30.160.20:38813 - accepting connections
I20260808 02:23:27.756610 31360 mini_ranger.cc:161] Starting Ranger...
I20260808 02:23:27.771209 31360 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:23:28.233400 31360 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:23:28.233477 31360 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-admin
2026-08-08 02:23:28,524 [I] DB FLAVOR :POSTGRES
2026-08-08 02:23:28,524 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:23:28,524 [I] Checking connection..
2026-08-08 02:23:28,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:23:28,787 [I] Checking connection passed.
2026-08-08 02:23:28,787 [I] --------- Verifying version history table ---------
2026-08-08 02:23:28,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:23:29,034 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:23:29,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:23:29,287 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:23:29,287 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:23:29,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:23:29.545 UTC [10145] WARNING: there is no transaction in progress
2026-08-08 02:23:29,552 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:23:29,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:23:29,804 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:23:29,804 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:23:29,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:23:30,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:23:30,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zkpd.c.gcp-upstream.internal','N') ;"
2026-08-08 02:23:30,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:23:30,803 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:23:30,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:23:31,069 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:23:31,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:23:31,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:23:31,582 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:23:31,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:23:32.503 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.519 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.534 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.549 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.565 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.582 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.643 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.649 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.663 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.674 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.684 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.697 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.706 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.712 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.719 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:32.725 UTC [10265] WARNING: there is no transaction in progress
2026-08-08 02:23:33,346 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:23:33,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:23:33,581 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:23:33,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:23:33,845 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:23:33,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:23:34,096 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:23:34,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:23:34,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:23:34,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zkpd.c.gcp-upstream.internal';"
2026-08-08 02:23:34,805 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:23:34,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:23:35,035 [I] DB_PATCHES have already been applied
I20260808 02:23:35.041663 31360 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-admin:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8DZQyQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:23:35.041729 31360 mini_ranger.cc:193] Using host: 127.30.160.20
I20260808 02:23:35.045596 31360 mini_ranger.cc:227] Ranger admin URL: http://127.30.160.20:41733
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:23:36.207151 31360 mini_ranger.cc:160] Time spent starting Ranger: real 8.451s user 0.001s sys 0.009s
Aug 08, 2026 2:23:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:23:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 2:23:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:23:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:23:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:23:58.584349 31360 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/rangerkms_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/rangerkms_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.160.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.160.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:23:58 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155838, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:23:58.675233 31360 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:23:58.686148 31360 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:23:59.486181 31360 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:23:59.486276 31360 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms
I20260808 02:23:59.486295 31360 mini_ranger_kms.cc:227] Using postgres at 127.30.160.20:38813
2026-08-08 02:23:59,674 [I] DB FLAVOR :POSTGRES
2026-08-08 02:23:59,674 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:23:59,674 [I] Checking connection
2026-08-08 02:23:59,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:23:59,916 [I] connection success
2026-08-08 02:23:59,917 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:23:59,917 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:23:59,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:24:00,147 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:24:00,147 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:24:00,147 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:24:00,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:38813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:24:00,415 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:24:00.926208 31360 mini_ranger_kms.cc:310] Created kms service
I20260808 02:24:01.095654 31360 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:24:01.141144 31360 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:24:01 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155838, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:01.529870 31360 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:24:01.529985 31360 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms:/tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:24:01.530005 31360 mini_ranger_kms.cc:240] Using host: 127.30.160.20
I20260808 02:24:01.534169 31360 mini_ranger_kms.cc:276] Ranger KMS PID: 10465
I20260808 02:24:01.534301 31360 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.160.20:50981
Aug 08, 2026 2:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:24:03 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.160.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:03.805689 31360 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.130s user 0.007s sys 0.161s
I20260808 02:24:03.805795 31360 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:24:03.805881 31360 mini_ranger_kms.cc:401] 127.30.160.20:50981/kms/v1/keys
Aug 08 02:24:04 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.160.20@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
Aug 08 02:24:04 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.160.20@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
Aug 08 02:24:06 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:24:06.902549 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39457
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39457
--ranger_config_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:50981/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:06.986052 10519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:06.986241 10519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:06.986263 10519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:06.987389 10519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:06.987432 10519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:06.987445 10519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:06.987481 10519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:06.987512 10519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:06.989089 10519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.160.20:50981/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39457
--ranger_config_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:06.989332 10519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:06.989560 10519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:06.994509 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:06.994500 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:06.994483 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:06.994694 10519 server_base.cc:1034] running on GCE node
I20260808 02:24:06.994973 10519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:06.995324 10519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:06.996484 10519 hybrid_clock.cc:648] HybridClock initialized: now 1786155846996472 us; error 26 us; skew 500 ppm
Aug 08 02:24:06 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:06.998857 10519 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:24:06.999289 10519 webserver.cc:460] Webserver started at http://127.30.160.62:40683/ using document root <none> and password file <none>
I20260808 02:24:06.999513 10519 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:06.999573 10519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:06.999732 10519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:06 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:07.047192 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "7c92fd969ba049af9033bdb22f6474f7"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-zkpd"
server_key: "dfc93f61bdcb25265be53b808323001a"
server_key_iv: "b7f9bd7328a1a8545c8f69c9012a50e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:07.047636 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "7c92fd969ba049af9033bdb22f6474f7"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-zkpd"
server_key: "dfc93f61bdcb25265be53b808323001a"
server_key_iv: "b7f9bd7328a1a8545c8f69c9012a50e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:07.049273 10519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:24:07.061105 10536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:07.061321 10519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:24:07.061425 10519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "7c92fd969ba049af9033bdb22f6474f7"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-zkpd"
server_key: "dfc93f61bdcb25265be53b808323001a"
server_key_iv: "b7f9bd7328a1a8545c8f69c9012a50e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:07.061534 10519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:07.102676 10519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:07.104910 10519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:07.105080 10519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:07.110543 10519 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39457
I20260808 02:24:07.110569 10590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39457 every 8 connection(s)
I20260808 02:24:07.110950 10519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:07.111850 10591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:07.115100 10591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7: Bootstrap starting.
I20260808 02:24:07.115841 10591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:07.116220 10591 log.cc:824] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:07.117013 10591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7: No bootstrap required, opened a new log
I20260808 02:24:07.118858 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 10519
I20260808 02:24:07.119000 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:24:07.119199 31360 external_mini_cluster.cc:1428] Setting key f5e3154b97e10f0c71cf11aaa9092a30
I20260808 02:24:07.120175 10591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c92fd969ba049af9033bdb22f6474f7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39457 } }
I20260808 02:24:07.120421 10591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:07.120477 10591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c92fd969ba049af9033bdb22f6474f7, State: Initialized, Role: FOLLOWER
I20260808 02:24:07.120625 10591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c92fd969ba049af9033bdb22f6474f7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39457 } }
I20260808 02:24:07.120707 10591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:07.120739 10591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:07.120818 10591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:07.121556 10591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c92fd969ba049af9033bdb22f6474f7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39457 } }
I20260808 02:24:07.121670 10591 leader_election.cc:302] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c92fd969ba049af9033bdb22f6474f7; no voters:
I20260808 02:24:07.121843 10591 leader_election.cc:288] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:07.121893 10596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:07.122193 10591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:07.122207 10596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [term 1 LEADER]: Becoming Leader. State: Replica: 7c92fd969ba049af9033bdb22f6474f7, State: Running, Role: LEADER
I20260808 02:24:07.122395 10596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c92fd969ba049af9033bdb22f6474f7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39457 } }
I20260808 02:24:07.122701 10597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c92fd969ba049af9033bdb22f6474f7. Latest consensus state: current_term: 1 leader_uuid: "7c92fd969ba049af9033bdb22f6474f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c92fd969ba049af9033bdb22f6474f7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39457 } } }
I20260808 02:24:07.122772 10597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:07.122978 10603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:07.123436 10603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:07.123799 10591 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:24:07.124096 10598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c92fd969ba049af9033bdb22f6474f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c92fd969ba049af9033bdb22f6474f7" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39457 } } }
I20260808 02:24:07.124238 10598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7 [sys.catalog]: This master's current role is: LEADER
Aug 08 02:24:07 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:07.135527 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:07.119714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.30.160.62:39457)
0808 02:24:07.119926 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:07.119937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:07.119972 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:24:07.120123 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:07.120129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:07.120173 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:07.120309 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:07.121113 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.122396 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.123671 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.124376 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.126622 (+ 2246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:07.130218 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:07.130223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:07.130223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:07.130234 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:07.132878 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:07.133308 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:07.133311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:07.133311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:07.133442 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:07.133668 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:07.133669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:07.133669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:07.134042 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:07.134832 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:07.134915 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 02:24:07.135189 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260808 02:24:07.136586 10603 catalog_manager.cc:1347] Generated new cluster ID: 4011e2979b3948ab8332418ad4cefc02
I20260808 02:24:07.136675 10603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:07.159569 10603 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:07.160434 10603 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:07.166898 10603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7: Generated new TSK 0
I20260808 02:24:07.167145 10603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:24:07.825125 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:50981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39457
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:07.942952 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:07.943239 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:07.943279 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:07.945036 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:07.945096 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:07.945137 10638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:24:07.947052 10638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.160.20:50981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:07.947281 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:07.947542 10638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:07.953656 10646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:07.953917 10638 server_base.cc:1034] running on GCE node
W20260808 02:24:07.953748 10644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:07.953675 10643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:07.954793 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:07.955046 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:07.956209 10638 hybrid_clock.cc:648] HybridClock initialized: now 1786155847956197 us; error 31 us; skew 500 ppm
Aug 08 02:24:07 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155847, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:07.959141 10638 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:24:07.959656 10638 webserver.cc:460] Webserver started at http://127.30.160.1:40809/ using document root <none> and password file <none>
I20260808 02:24:07.959964 10638 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:07.960048 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:07.960189 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:07 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155847, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:07.972946 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "a27c8098d01e4d8ab215f386ef8f66f6"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-zkpd"
server_key: "d61ee96abacf7a7f43dad3fbb94d5a07"
server_key_iv: "6e797fd2fdfff06b0fbb6c0aa3891377"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:07.973392 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "a27c8098d01e4d8ab215f386ef8f66f6"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-zkpd"
server_key: "d61ee96abacf7a7f43dad3fbb94d5a07"
server_key_iv: "6e797fd2fdfff06b0fbb6c0aa3891377"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:07.976259 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:24:07.988118 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:07.988356 10638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:24:07.988489 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "a27c8098d01e4d8ab215f386ef8f66f6"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-zkpd"
server_key: "d61ee96abacf7a7f43dad3fbb94d5a07"
server_key_iv: "6e797fd2fdfff06b0fbb6c0aa3891377"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:07.988612 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:08.038298 10638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:08.040911 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:08.041133 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:08.041453 10638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:08.041874 10638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:08.041962 10638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.042058 10638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:08.042131 10638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.049724 10638 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:39437
I20260808 02:24:08.049800 10775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:39437 every 8 connection(s)
I20260808 02:24:08.050122 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:08.052495 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 10638
I20260808 02:24:08.052636 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:08.052841 31360 external_mini_cluster.cc:1428] Setting key fc34c34090e5505569f0f9d19367702d
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155847, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:08.061197 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.050897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39457 (local address 127.30.160.1:36497)
0808 02:24:08.051078 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.051122 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.051784 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.052125 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.052130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.052380 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:08.052667 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.052673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.053652 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.053653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.054949 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.054952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.055459 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.055461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.055600 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.056178 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:08.056187 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:08.057469 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:08.059477 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.059484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.059486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.059772 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.060031 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.060032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.060032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.060181 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.060509 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:08.060516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:08.060679 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.060795 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.060955 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260808 02:24:08.061757 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.050961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:36497 (local address 127.30.160.62:39457)
0808 02:24:08.051110 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.051112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.051197 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:24:08.051945 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.051946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.051963 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.052030 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:08.052764 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.053475 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.055053 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.055362 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.055450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.057564 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:08.057566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:08.057567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:08.057574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:08.059382 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.059862 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.059863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.059864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.059956 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.060261 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.060262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.060263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.060414 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:08.060479 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.061564 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.061658 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260808 02:24:08.062541 10777 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39457
I20260808 02:24:08.062690 10777 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:08.062935 10777 heartbeater.cc:511] Master 127.30.160.62:39457 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
I20260808 02:24:08.063535 10555 ts_manager.cc:194] Registered new tserver with Master: a27c8098d01e4d8ab215f386ef8f66f6 (127.30.160.1:39437)
I20260808 02:24:08.064692 10555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:36497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:24:08.083981 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:50981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39457
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:24:08.204241 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:08.204576 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:08.204655 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:08.206341 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:08.206444 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:08.206530 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:24:08.208714 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.160.20:50981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:08.208962 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:08.209252 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:08.215421 10795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.215597 10793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.215683 10792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:08.216732 10785 server_base.cc:1034] running on GCE node
I20260808 02:24:08.216974 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:08.217224 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:08.220515 10785 hybrid_clock.cc:648] HybridClock initialized: now 1786155848220481 us; error 44 us; skew 500 ppm
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:08.223729 10785 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:24:08.224257 10785 webserver.cc:460] Webserver started at http://127.30.160.2:36145/ using document root <none> and password file <none>
I20260808 02:24:08.224506 10785 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:08.224558 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:08.224678 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:08.237548 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "ff2aafe1763d4a8d877c69c9d7f3320e"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-zkpd"
server_key: "12ecfb612da257d7fe0402c9644fdf1a"
server_key_iv: "0dbcb631341981c69c34b27582327f27"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:08.238148 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "ff2aafe1763d4a8d877c69c9d7f3320e"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-zkpd"
server_key: "12ecfb612da257d7fe0402c9644fdf1a"
server_key_iv: "0dbcb631341981c69c34b27582327f27"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:08.244293 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260808 02:24:08.260030 10802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.260331 10785 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260808 02:24:08.260447 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "ff2aafe1763d4a8d877c69c9d7f3320e"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-zkpd"
server_key: "12ecfb612da257d7fe0402c9644fdf1a"
server_key_iv: "0dbcb631341981c69c34b27582327f27"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:08.260541 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:08.274421 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:08.276844 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:08.277024 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:08.277314 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:08.277762 10785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:08.277840 10785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.277902 10785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:08.277966 10785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.285475 10785 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:43851
I20260808 02:24:08.285949 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:08.290478 10922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:43851 every 8 connection(s)
I20260808 02:24:08.292371 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 10785
I20260808 02:24:08.292479 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:08.292665 31360 external_mini_cluster.cc:1428] Setting key 38c6d14b07887dfdd42e28e34e65f530
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:08.302205 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.291229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:42161 (local address 127.30.160.62:39457)
0808 02:24:08.291485 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.291487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.291562 (+ 75us) server_negotiation.cc:407] Connection header received
0808 02:24:08.292305 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.292307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.292323 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.292356 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:08.293244 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.294051 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.295633 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.295965 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.296010 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.298640 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:08.298642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:08.298642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:08.298650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:08.300502 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.301018 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.301019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.301020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.301085 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.301423 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.301424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.301424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.301566 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:08.301660 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.302014 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.302106 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260808 02:24:08.302416 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.291218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39457 (local address 127.30.160.2:42161)
0808 02:24:08.291422 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.291470 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.292172 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.292598 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.292603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.292838 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:08.293118 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.293124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.294239 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.294240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.295529 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.295532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.296100 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.296101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.296243 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.296866 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:08.296876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:08.298533 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:08.300598 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.300604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.300606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.300929 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.301169 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.301170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.301170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.301340 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.301664 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:08.301670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:08.301846 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.301974 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.302135 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260808 02:24:08.303057 10923 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39457
I20260808 02:24:08.303166 10923 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:08.303382 10923 heartbeater.cc:511] Master 127.30.160.62:39457 requested a full tablet report, sending...
I20260808 02:24:08.303824 10555 ts_manager.cc:194] Registered new tserver with Master: ff2aafe1763d4a8d877c69c9d7f3320e (127.30.160.2:43851)
I20260808 02:24:08.304334 10555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:42161
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:24:08.326920 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:50981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39457
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:08.450726 10928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:08.451054 10928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:08.451099 10928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:08.453225 10928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:08.453296 10928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:08.453346 10928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:24:08.455755 10928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.160.20:50981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:08.456244 10928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:08.456575 10928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:08.469380 10934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.469483 10933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.469537 10936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:08.470527 10928 server_base.cc:1034] running on GCE node
I20260808 02:24:08.470887 10928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:08.471201 10928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:08.472368 10928 hybrid_clock.cc:648] HybridClock initialized: now 1786155848472348 us; error 45 us; skew 500 ppm
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:08.475373 10928 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:24:08.475893 10928 webserver.cc:460] Webserver started at http://127.30.160.3:37355/ using document root <none> and password file <none>
I20260808 02:24:08.476186 10928 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:08.476257 10928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:08.476406 10928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:08.489564 10928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "357dd9b6f8fe4e09baa3693a18ec111b"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-zkpd"
server_key: "06fae3dfe7be0705fc97ac98ec6d7557"
server_key_iv: "dd6ad227a2a8c16e538fe0f4aec5c13e"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:08.490010 10928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "357dd9b6f8fe4e09baa3693a18ec111b"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-zkpd"
server_key: "06fae3dfe7be0705fc97ac98ec6d7557"
server_key_iv: "dd6ad227a2a8c16e538fe0f4aec5c13e"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:08.492013 10928 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:24:08.512979 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.513175 10928 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260808 02:24:08.513285 10928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "357dd9b6f8fe4e09baa3693a18ec111b"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-zkpd"
server_key: "06fae3dfe7be0705fc97ac98ec6d7557"
server_key_iv: "dd6ad227a2a8c16e538fe0f4aec5c13e"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:08.513377 10928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:08.527469 10928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:08.530017 10928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:08.530190 10928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:08.530465 10928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:08.530869 10928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:08.530956 10928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.531026 10928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:08.531078 10928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.541990 10928 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:34179
I20260808 02:24:08.542531 10928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:08.542907 11065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:34179 every 8 connection(s)
I20260808 02:24:08.544673 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 10928
I20260808 02:24:08.544792 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:08.544988 31360 external_mini_cluster.cc:1428] Setting key 2cd0c9f5cd942d2fd6bd86b2c6475f7d
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:08.555025 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.543663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:47169 (local address 127.30.160.62:39457)
0808 02:24:08.543720 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.543722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.544064 (+ 342us) server_negotiation.cc:407] Connection header received
0808 02:24:08.544889 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.544891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.544905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.544939 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:08.546167 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.546930 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.548493 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.548807 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.548887 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.551496 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:08.551497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:08.551497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:08.551505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:08.553383 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.553897 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.553898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.553899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.553968 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.554278 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.554279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.554279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.554419 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:08.554495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.554838 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.554924 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:24:08.555202 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.543620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39457 (local address 127.30.160.3:47169)
0808 02:24:08.543921 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.543968 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.544751 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.545208 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.545213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.545587 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:08.546057 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.546065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.547052 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.547053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.548370 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.548374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.548916 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.548917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.549065 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.549674 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:08.549686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:08.551377 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:08.553484 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.553491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.553493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.553808 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.554053 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.554054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.554054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.554197 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.554504 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:08.554512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:08.554674 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.554798 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.554957 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260808 02:24:08.555910 11067 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39457
I20260808 02:24:08.556020 11067 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:08.556445 11067 heartbeater.cc:511] Master 127.30.160.62:39457 requested a full tablet report, sending...
I20260808 02:24:08.557751 10555 ts_manager.cc:194] Registered new tserver with Master: 357dd9b6f8fe4e09baa3693a18ec111b (127.30.160.3:34179)
I20260808 02:24:08.558127 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:24:08.558351 10555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:47169
I20260808 02:24:08.571882 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.559823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.30.160.62:39457)
0808 02:24:08.559867 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.559869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.563879 (+ 4010us) server_negotiation.cc:407] Connection header received
0808 02:24:08.564011 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.564012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.564023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.564052 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:08.564924 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.565709 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.566869 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.567145 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.567189 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.568817 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:08.568818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:08.568819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:08.568826 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:08.570535 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.571011 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.571012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.571013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.571077 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.571316 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.571316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.571317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.571446 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:08.571547 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.571706 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.571786 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:24:08.573726 10555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:08.576419 10555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:08.588866 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.579701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34179 (local address 127.0.0.1:47384)
0808 02:24:08.580737 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.580781 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.580848 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.581865 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.581866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.581868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:08.581900 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.581901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.585673 (+ 3772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.585673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.586723 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.586725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.588595 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.588595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.588673 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.588685 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.588750 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.588795 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":76,"threads_started":1}
I20260808 02:24:08.589483 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.580018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39437 (local address 127.0.0.1:57812)
0808 02:24:08.581301 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.581312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.581369 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.581659 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.581663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.581666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:08.581735 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.581737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.584369 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.584370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.585580 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.585583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.589287 (+ 3704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.589288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.589346 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.589349 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.589386 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.589420 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"thread_start_us":69,"threads_started":1}
I20260808 02:24:08.590062 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.580471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.30.160.1:39437)
0808 02:24:08.581249 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.581253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.581431 (+ 178us) server_negotiation.cc:407] Connection header received
0808 02:24:08.581475 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.581478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.581497 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.581574 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:08.583201 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.584276 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.586819 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.587539 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.587607 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.587663 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.589943 (+ 2280us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.589990 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":46,"threads_started":1}
I20260808 02:24:08.590365 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.580313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.30.160.3:34179)
0808 02:24:08.580986 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.580991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.581001 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:24:08.581038 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.581041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.581056 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.581126 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:08.581985 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.583099 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.587714 (+ 4615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.588431 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.588496 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.588560 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.590253 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.590292 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":84,"threads_started":1}
I20260808 02:24:08.591123 10697 tablet_service.cc:1467] Processing CreateTablet for tablet e54ec80c40a3490da6e7df5160072022 (DEFAULT_TABLE table=test-table [id=27716060f77544ca8c7359761d9596e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:08.591493 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54ec80c40a3490da6e7df5160072022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:08.595624 10988 tablet_service.cc:1467] Processing CreateTablet for tablet e54ec80c40a3490da6e7df5160072022 (DEFAULT_TABLE table=test-table [id=27716060f77544ca8c7359761d9596e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:08.596032 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54ec80c40a3490da6e7df5160072022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:08.597044 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.580056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.30.160.2:43851)
0808 02:24:08.586376 (+ 6320us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.586381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.591970 (+ 5589us) server_negotiation.cc:407] Connection header received
0808 02:24:08.592025 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.592028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.592046 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.592134 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:08.592771 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.593968 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.595378 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.596169 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.596245 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.596337 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.596919 (+ 582us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.596961 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6274,"thread_start_us":91,"threads_started":1}
I20260808 02:24:08.597155 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.580575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43851 (local address 127.0.0.1:33758)
0808 02:24:08.589152 (+ 8577us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.589162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.589230 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.592228 (+ 2998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.592229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.592231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:08.592268 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.592269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.594063 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.594064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.595243 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.595247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.596377 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.596379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.596439 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.596443 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.596498 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.596535 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8560,"thread_start_us":83,"threads_started":1}
I20260808 02:24:08.597770 10854 tablet_service.cc:1467] Processing CreateTablet for tablet e54ec80c40a3490da6e7df5160072022 (DEFAULT_TABLE table=test-table [id=27716060f77544ca8c7359761d9596e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:08.598097 10854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54ec80c40a3490da6e7df5160072022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:08.599481 11085 tablet_bootstrap.cc:492] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6: Bootstrap starting.
I20260808 02:24:08.600232 11085 tablet_bootstrap.cc:654] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:08.600427 11086 tablet_bootstrap.cc:492] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Bootstrap starting.
I20260808 02:24:08.600808 11085 log.cc:824] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:08.601617 11087 tablet_bootstrap.cc:492] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Bootstrap starting.
I20260808 02:24:08.601753 11086 tablet_bootstrap.cc:654] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:08.602022 11086 log.cc:824] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:08.603101 11087 tablet_bootstrap.cc:654] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:08.603152 11086 tablet_bootstrap.cc:492] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: No bootstrap required, opened a new log
I20260808 02:24:08.603266 11086 ts_tablet_manager.cc:1397] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:08.603526 11087 log.cc:824] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:08.604344 11087 tablet_bootstrap.cc:492] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: No bootstrap required, opened a new log
I20260808 02:24:08.604411 11087 ts_tablet_manager.cc:1397] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:08.605296 11086 raft_consensus.cc:348] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.605450 11086 raft_consensus.cc:374] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:08.605480 11086 raft_consensus.cc:729] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357dd9b6f8fe4e09baa3693a18ec111b, State: Initialized, Role: FOLLOWER
I20260808 02:24:08.605615 11086 consensus_queue.cc:260] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.605859 11086 ts_tablet_manager.cc:1428] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:08.605968 11067 heartbeater.cc:503] Master 127.30.160.62:39457 was elected leader, sending a full tablet report...
I20260808 02:24:08.606559 11085 tablet_bootstrap.cc:492] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6: No bootstrap required, opened a new log
I20260808 02:24:08.606456 11087 raft_consensus.cc:348] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.606597 11087 raft_consensus.cc:374] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:08.606616 11085 ts_tablet_manager.cc:1397] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260808 02:24:08.606631 11087 raft_consensus.cc:729] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2aafe1763d4a8d877c69c9d7f3320e, State: Initialized, Role: FOLLOWER
I20260808 02:24:08.606750 11087 consensus_queue.cc:260] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.606992 11087 ts_tablet_manager.cc:1428] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:08.607993 10923 heartbeater.cc:503] Master 127.30.160.62:39457 was elected leader, sending a full tablet report...
I20260808 02:24:08.609469 11085 raft_consensus.cc:348] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.609630 11085 raft_consensus.cc:374] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:08.609659 11085 raft_consensus.cc:729] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27c8098d01e4d8ab215f386ef8f66f6, State: Initialized, Role: FOLLOWER
I20260808 02:24:08.609810 11085 consensus_queue.cc:260] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.610080 11085 ts_tablet_manager.cc:1428] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 02:24:08.610128 10777 heartbeater.cc:503] Master 127.30.160.62:39457 was elected leader, sending a full tablet report...
Aug 08 02:24:08 dist-test-slave-zkpd krb5kdc[10034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:08.697364 11093 raft_consensus.cc:482] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:08.697526 11093 raft_consensus.cc:504] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.697973 11093 leader_election.cc:288] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357dd9b6f8fe4e09baa3693a18ec111b (127.30.160.3:34179), ff2aafe1763d4a8d877c69c9d7f3320e (127.30.160.2:43851)
I20260808 02:24:08.702807 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.698812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:37555 (local address 127.30.160.3:34179)
0808 02:24:08.698935 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.698938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.699131 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:24:08.699247 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.699248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.699259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.699292 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:08.699549 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.700337 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.701841 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.702410 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.702462 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.702472 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.702695 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.702733 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 02:24:08.703362 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.698379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:34179 (local address 127.30.160.1:37555)
0808 02:24:08.699058 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.699070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.699162 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.699370 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.699371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.699373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:08.699462 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.699464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.700469 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.700470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.701721 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.701724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.702536 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.702537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.702612 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.702615 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.703229 (+ 614us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.703276 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":100,"threads_started":1}
I20260808 02:24:08.703847 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54ec80c40a3490da6e7df5160072022" candidate_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" is_pre_election: true
I20260808 02:24:08.704037 11016 raft_consensus.cc:2454] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a27c8098d01e4d8ab215f386ef8f66f6 in term 0.
I20260808 02:24:08.704289 10657 leader_election.cc:302] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357dd9b6f8fe4e09baa3693a18ec111b, a27c8098d01e4d8ab215f386ef8f66f6; no voters:
I20260808 02:24:08.704419 11093 raft_consensus.cc:2781] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:08.704466 11093 raft_consensus.cc:482] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:08.704487 11093 raft_consensus.cc:3037] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:08.705452 11093 raft_consensus.cc:504] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.705580 11093 leader_election.cc:288] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [CANDIDATE]: Term 1 election: Requested vote from peers 357dd9b6f8fe4e09baa3693a18ec111b (127.30.160.3:34179), ff2aafe1763d4a8d877c69c9d7f3320e (127.30.160.2:43851)
I20260808 02:24:08.705920 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54ec80c40a3490da6e7df5160072022" candidate_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357dd9b6f8fe4e09baa3693a18ec111b"
I20260808 02:24:08.705996 11016 raft_consensus.cc:3037] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:08.706893 11016 raft_consensus.cc:2454] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a27c8098d01e4d8ab215f386ef8f66f6 in term 1.
I20260808 02:24:08.707120 10657 leader_election.cc:302] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357dd9b6f8fe4e09baa3693a18ec111b, a27c8098d01e4d8ab215f386ef8f66f6; no voters:
I20260808 02:24:08.707242 11093 raft_consensus.cc:2781] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:08.707386 11093 raft_consensus.cc:686] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [term 1 LEADER]: Becoming Leader. State: Replica: a27c8098d01e4d8ab215f386ef8f66f6, State: Running, Role: LEADER
I20260808 02:24:08.707474 11093 consensus_queue.cc:237] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } }
I20260808 02:24:08.708341 10553 catalog_manager.cc:5495] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27c8098d01e4d8ab215f386ef8f66f6 (127.30.160.1). New cstate: current_term: 1 leader_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260808 02:24:08.712539 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.698884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:37081 (local address 127.30.160.2:43851)
0808 02:24:08.698971 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.698973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.708910 (+ 9937us) server_negotiation.cc:407] Connection header received
0808 02:24:08.709021 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.709022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.709037 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.709072 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:08.709493 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.710251 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.711599 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.712170 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.712220 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.712230 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.712430 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.712469 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:24:08.712895 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.698591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43851 (local address 127.30.160.1:37081)
0808 02:24:08.708837 (+ 10246us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.708852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.708941 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.709371 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.709372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.709374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:08.709405 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.709407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.710376 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.710377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.711452 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.711455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.712289 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.712289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.712348 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.712351 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.712776 (+ 425us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.712817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10221,"thread_start_us":73,"threads_started":1}
I20260808 02:24:08.713349 10872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54ec80c40a3490da6e7df5160072022" candidate_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" is_pre_election: true
I20260808 02:24:08.713517 10872 raft_consensus.cc:2454] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a27c8098d01e4d8ab215f386ef8f66f6 in term 0.
I20260808 02:24:08.713653 10875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54ec80c40a3490da6e7df5160072022" candidate_uuid: "a27c8098d01e4d8ab215f386ef8f66f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e"
I20260808 02:24:08.713748 10875 raft_consensus.cc:3037] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:08.714704 10875 raft_consensus.cc:2454] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a27c8098d01e4d8ab215f386ef8f66f6 in term 1.
I20260808 02:24:08.731707 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.728450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.30.160.1:39437)
0808 02:24:08.728491 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.728493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.728499 (+ 6us) server_negotiation.cc:407] Connection header received
0808 02:24:08.728534 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.728535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.728553 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.728612 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:08.728766 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.729491 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.730632 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.730927 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.730991 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.731184 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:08.731310 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:08.731406 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.731614 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.731633 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:24:08.734295 10875 raft_consensus.cc:1261] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 1 FOLLOWER]: Refusing update from remote peer a27c8098d01e4d8ab215f386ef8f66f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:08.734707 11098 consensus_queue.cc:1035] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2aafe1763d4a8d877c69c9d7f3320e" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:08.737023 11016 raft_consensus.cc:1261] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 1 FOLLOWER]: Refusing update from remote peer a27c8098d01e4d8ab215f386ef8f66f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:08.737351 11098 consensus_queue.cc:1035] T e54ec80c40a3490da6e7df5160072022 P a27c8098d01e4d8ab215f386ef8f66f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "357dd9b6f8fe4e09baa3693a18ec111b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:08.740399 11104 mvcc.cc:205] Tried to move back new op lower bound from 7316094356412473344 to 7316094356306374656. Current Snapshot: MvccSnapshot[applied={T|T < 7316094356412473344}]
I20260808 02:24:08.740748 11102 mvcc.cc:205] Tried to move back new op lower bound from 7316094356412473344 to 7316094356306374656. Current Snapshot: MvccSnapshot[applied={T|T < 7316094356412473344}]
I20260808 02:24:08.741765 11101 mvcc.cc:205] Tried to move back new op lower bound from 7316094356412473344 to 7316094356306374656. Current Snapshot: MvccSnapshot[applied={T|T < 7316094356412473344 or (T in {7316094356412473344})}]
I20260808 02:24:08.743363 10553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:08.743454 10553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:08.744256 10553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c92fd969ba049af9033bdb22f6474f7: Sending DeleteTablet for 3 replicas of tablet e54ec80c40a3490da6e7df5160072022
I20260808 02:24:08.744702 10988 tablet_service.cc:1514] Processing DeleteTablet for tablet e54ec80c40a3490da6e7df5160072022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:08 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:47384
I20260808 02:24:08.744830 10854 tablet_service.cc:1514] Processing DeleteTablet for tablet e54ec80c40a3490da6e7df5160072022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:08 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:33758
I20260808 02:24:08.744920 10697 tablet_service.cc:1514] Processing DeleteTablet for tablet e54ec80c40a3490da6e7df5160072022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:08 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:57812
I20260808 02:24:08.745155 11111 tablet_replica.cc:330] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: stopping tablet replica
I20260808 02:24:08.745191 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 10638
I20260808 02:24:08.745281 11111 raft_consensus.cc:2229] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:08.745383 11111 raft_consensus.cc:2258] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:08.746147 11111 ts_tablet_manager.cc:1905] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:08.747180 11112 tablet_replica.cc:330] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: stopping tablet replica
I20260808 02:24:08.747342 11112 raft_consensus.cc:2229] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:08.747496 11112 raft_consensus.cc:2258] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:08.748668 11112 ts_tablet_manager.cc:1905] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:08.748811 11111 ts_tablet_manager.cc:1918] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:08.748931 11111 log.cc:1196] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/e54ec80c40a3490da6e7df5160072022
I20260808 02:24:08.749567 11111 ts_tablet_manager.cc:1939] T e54ec80c40a3490da6e7df5160072022 P 357dd9b6f8fe4e09baa3693a18ec111b: Deleting consensus metadata
I20260808 02:24:08.750069 10537 catalog_manager.cc:4841] TS 357dd9b6f8fe4e09baa3693a18ec111b (127.30.160.3:34179): tablet e54ec80c40a3490da6e7df5160072022 (table test-table [id=27716060f77544ca8c7359761d9596e4]) successfully deleted
I20260808 02:24:08.750554 11112 ts_tablet_manager.cc:1918] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:08.750651 11112 log.cc:1196] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/e54ec80c40a3490da6e7df5160072022
I20260808 02:24:08.751145 11112 ts_tablet_manager.cc:1939] T e54ec80c40a3490da6e7df5160072022 P ff2aafe1763d4a8d877c69c9d7f3320e: Deleting consensus metadata
I20260808 02:24:08.751564 10537 catalog_manager.cc:4841] TS ff2aafe1763d4a8d877c69c9d7f3320e (127.30.160.2:43851): tablet e54ec80c40a3490da6e7df5160072022 (table test-table [id=27716060f77544ca8c7359761d9596e4]) successfully deleted
W20260808 02:24:08.753743 10539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:39437 (error 108)
I20260808 02:24:08.754040 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 10785
I20260808 02:24:08.755388 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.754254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39437 (local address 127.0.0.1:57824)
0808 02:24:08.755319 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.755361 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1044}
W20260808 02:24:08.755563 10539 catalog_manager.cc:4584] TS a27c8098d01e4d8ab215f386ef8f66f6 (127.30.160.1:39437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e54ec80c40a3490da6e7df5160072022: Network error: Client connection negotiation failed: client connection to 127.30.160.1:39437: connect: Connection refused (error 111)
I20260808 02:24:08.760783 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 10928
I20260808 02:24:08.766881 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 10519
I20260808 02:24:08.773641 31360 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:24:09.118036 31360 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:24:09.118168 31360 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:24:09.472831 31360 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:24:09.473 UTC [10065] LOG: received smart shutdown request
2026-08-08 02:24:09.476 UTC [10065] LOG: background worker "logical replication launcher" (PID 10076) exited with exit code 1
2026-08-08 02:24:09.476 UTC [10070] LOG: shutting down
2026-08-08 02:24:09.628 UTC [10065] LOG: database system is shut down
2026-08-08T02:24:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46766 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:09 dist-test-slave-zkpd krb5kdc[11123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:09 dist-test-slave-zkpd krb5kdc[11123](info): set up 2 sockets
Aug 08 02:24:09 dist-test-slave-zkpd krb5kdc[11123](info): commencing operation
krb5kdc: starting...
W20260808 02:24:10.831826 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:24:10 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/rangeradmin_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/rangeradmin_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/rangerlookup_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/rangerlookup_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
I20260808 02:24:10.994138 31360 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8DZQyQ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/postgres -l logfile start
2026-08-08 02:24:13.374 UTC [11157] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:24:13.374 UTC [11157] LOG: listening on IPv4 address "127.30.160.20", port 37501
2026-08-08 02:24:13.377 UTC [11157] LOG: listening on Unix socket "/tmp/.s.PGSQL.37501"
2026-08-08 02:24:13.397 UTC [11160] LOG: database system was shut down at 2026-08-08 02:24:12 UTC
2026-08-08 02:24:13.401 UTC [11157] LOG: database system is ready to accept connections
I20260808 02:24:14.387703 31360 mini_postgres.cc:96] Postgres bound to 37501
2026-08-08 02:24:14.394 UTC [11169] FATAL: database "slave" does not exist
127.30.160.20:37501 - accepting connections
I20260808 02:24:14.395117 31360 mini_ranger.cc:161] Starting Ranger...
I20260808 02:24:14.409700 31360 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:24:14.840637 31360 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:24:14.840745 31360 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-admin
2026-08-08 02:24:15,120 [I] DB FLAVOR :POSTGRES
2026-08-08 02:24:15,120 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:24:15,120 [I] Checking connection..
2026-08-08 02:24:15,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:24:15,363 [I] Checking connection passed.
2026-08-08 02:24:15,363 [I] --------- Verifying version history table ---------
2026-08-08 02:24:15,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:24:15,610 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:24:15,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:24:15,873 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:24:15,873 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:24:15,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:24:16.120 UTC [11237] WARNING: there is no transaction in progress
2026-08-08 02:24:16,127 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:24:16,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:24:16,364 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:24:16,364 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:24:16,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:24:16,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:24:16,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zkpd.c.gcp-upstream.internal','N') ;"
2026-08-08 02:24:17,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:24:17,353 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:24:17,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:24:17,603 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:24:17,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:24:17,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:24:18,068 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:24:18,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:24:18.971 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:18.986 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.002 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.017 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.033 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.049 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.102 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.108 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.123 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.133 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.143 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.157 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.166 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.173 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.179 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19.186 UTC [11357] WARNING: there is no transaction in progress
2026-08-08 02:24:19,810 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:24:19,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:24:20,031 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:24:20,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:24:20,263 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:24:20,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:24:20,527 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:24:20,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:24:20,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:24:20,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zkpd.c.gcp-upstream.internal';"
2026-08-08 02:24:21,246 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:24:21,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:24:21,486 [I] DB_PATCHES have already been applied
I20260808 02:24:21.492607 31360 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-admin:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8DZQyQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:24:21.492697 31360 mini_ranger.cc:193] Using host: 127.30.160.20
I20260808 02:24:21.496541 31360 mini_ranger.cc:227] Ranger admin URL: http://127.30.160.20:41423
Aug 08, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:24:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:24:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:24:22.635849 31360 mini_ranger.cc:160] Time spent starting Ranger: real 8.241s user 0.001s sys 0.008s
Aug 08, 2026 2:24:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:24:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 2:24:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:24:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:24:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:24:44.976336 31360 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/rangerkms_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/rangerkms_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.160.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.160.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.160.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/HTTP_127.30.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:24:45 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155885, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:24:45.061465 31360 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:24:45.070602 31360 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:24:45.879482 31360 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:24:45.879558 31360 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms
I20260808 02:24:45.879571 31360 mini_ranger_kms.cc:227] Using postgres at 127.30.160.20:37501
2026-08-08 02:24:46,045 [I] DB FLAVOR :POSTGRES
2026-08-08 02:24:46,045 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:24:46,045 [I] Checking connection
2026-08-08 02:24:46,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:24:46,275 [I] connection success
2026-08-08 02:24:46,275 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:24:46,276 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:24:46,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:24:46,527 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:24:46,527 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:24:46,527 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:24:46,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.160.20:37501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:24:46,791 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:24:47.287842 31360 mini_ranger_kms.cc:310] Created kms service
I20260808 02:24:47.481267 31360 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:24:47.525744 31360 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:24:47 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155885, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:47.938181 31360 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:24:47.938294 31360 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms:/tmp/dist-test-task8DZQyQ/build/release/bin/postgresql.jar:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8DZQyQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:24:47.938314 31360 mini_ranger_kms.cc:240] Using host: 127.30.160.20
I20260808 02:24:47.943081 31360 mini_ranger_kms.cc:276] Ranger KMS PID: 11548
I20260808 02:24:47.943168 31360 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.160.20:57071
Aug 08, 2026 2:24:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:24:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:24:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:24:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:24:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:24:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 02:24:50.220466 31360 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.159s user 0.000s sys 0.149s
I20260808 02:24:50.220631 31360 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:24:50.220692 31360 mini_ranger_kms.cc:401] 127.30.160.20:57071/kms/v1/keys
Aug 08 02:24:50 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.160.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 02:24:50 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.160.20@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
Aug 08 02:24:51 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.160.20@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
Aug 08 02:24:53 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:24:53.306394 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40415
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:40415
--ranger_config_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:53.388342 11606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:53.388540 11606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:53.388562 11606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:53.389703 11606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:53.389745 11606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:53.389760 11606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:53.389771 11606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:53.389789 11606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:53.391412 11606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.160.20:57071/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:40415
--ranger_config_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:53.391649 11606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:53.391928 11606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:53.396881 11612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:53.396983 11606 server_base.cc:1034] running on GCE node
W20260808 02:24:53.396931 11611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:53.397033 11614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:53.397341 11606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:53.397667 11606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:53.398855 11606 hybrid_clock.cc:648] HybridClock initialized: now 1786155893398831 us; error 38 us; skew 500 ppm
Aug 08 02:24:53 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155893, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:53.401686 11606 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:24:53.402117 11606 webserver.cc:460] Webserver started at http://127.30.160.62:43217/ using document root <none> and password file <none>
I20260808 02:24:53.402347 11606 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:53.402413 11606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:53.402571 11606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:53 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155893, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:53.456800 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "14eda6e9881b4d5f96c20c713df9ba83"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-zkpd"
server_key: "0a2deec03765c565f814419a59ba90e2"
server_key_iv: "c793b565c10f4b5a5ed3e18ffbeacbfa"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:53.457319 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "14eda6e9881b4d5f96c20c713df9ba83"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-zkpd"
server_key: "0a2deec03765c565f814419a59ba90e2"
server_key_iv: "c793b565c10f4b5a5ed3e18ffbeacbfa"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:53.459318 11606 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:24:53.476330 11622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:53.476580 11606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:24:53.476753 11606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "14eda6e9881b4d5f96c20c713df9ba83"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-zkpd"
server_key: "0a2deec03765c565f814419a59ba90e2"
server_key_iv: "c793b565c10f4b5a5ed3e18ffbeacbfa"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:53.476994 11606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:53.501209 11606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:53.503345 11606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:53.503520 11606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:53.508765 11606 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:40415
I20260808 02:24:53.508798 11678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:40415 every 8 connection(s)
I20260808 02:24:53.509254 11606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:53.509984 11679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:53.512368 11679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Bootstrap starting.
I20260808 02:24:53.512593 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 11606
I20260808 02:24:53.512684 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:24:53.512907 31360 external_mini_cluster.cc:1428] Setting key 2007c4ea1d4fef4fd23e6bb07390bac8
I20260808 02:24:53.513139 11679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:53.513489 11679 log.cc:824] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:53.514425 11679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: No bootstrap required, opened a new log
I20260808 02:24:53.516230 11679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:53.516412 11679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:53.516448 11679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eda6e9881b4d5f96c20c713df9ba83, State: Initialized, Role: FOLLOWER
I20260808 02:24:53.516609 11679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:53.516691 11679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:53.516733 11679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:53.516800 11679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:53.517416 11679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:53.517541 11679 leader_election.cc:302] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14eda6e9881b4d5f96c20c713df9ba83; no voters:
I20260808 02:24:53.517750 11679 leader_election.cc:288] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:53.517778 11685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:53.518098 11679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:53.518550 11685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 LEADER]: Becoming Leader. State: Replica: 14eda6e9881b4d5f96c20c713df9ba83, State: Running, Role: LEADER
I20260808 02:24:53.518728 11685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:53.519138 11685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14eda6e9881b4d5f96c20c713df9ba83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } } }
I20260808 02:24:53.519291 11685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:53.519503 11685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14eda6e9881b4d5f96c20c713df9ba83. Latest consensus state: current_term: 1 leader_uuid: "14eda6e9881b4d5f96c20c713df9ba83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } } }
I20260808 02:24:53.519623 11685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:53.519793 11694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:53.520323 11694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 02:24:53 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:53.522061 11694 catalog_manager.cc:1347] Generated new cluster ID: a200ba4f34fc4800a5ca96bd9718ebed
I20260808 02:24:53.522109 11694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:53.525646 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:53.513424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.30.160.62:40415)
0808 02:24:53.513592 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:53.513601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:53.513631 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:24:53.514178 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:53.514187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:53.514249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:53.514410 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:53.515186 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:53.516966 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:53.518010 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:53.518657 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:53.518766 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:53.521592 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:53.521596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:53.521597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:53.521608 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:53.523520 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:53.523954 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:53.523957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:53.523957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:53.524077 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:53.524290 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:53.524291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:53.524291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:53.524608 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:53.524763 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:53.525305 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 02:24:53.525429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260808 02:24:53.526307 11679 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:24:53.536023 11694 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:53.537209 11694 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:53.544334 11694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Generated new TSK 0
I20260808 02:24:53.544598 11694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:24:54.294180 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40415
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:54.384200 11726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:54.384400 11726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:54.384423 11726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:54.385545 11726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:54.385612 11726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:54.385668 11726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:24:54.387135 11726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.160.20:57071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:54.387385 11726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:54.387647 11726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:54.393630 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.393644 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.393914 11735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.393968 11726 server_base.cc:1034] running on GCE node
I20260808 02:24:54.394233 11726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:54.394490 11726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:54.395650 11726 hybrid_clock.cc:648] HybridClock initialized: now 1786155894395633 us; error 35 us; skew 500 ppm
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:54.398249 11726 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:24:54.398658 11726 webserver.cc:460] Webserver started at http://127.30.160.1:33497/ using document root <none> and password file <none>
I20260808 02:24:54.398895 11726 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:54.398976 11726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:54.399116 11726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:54.412912 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "f8babf411daa0381186ac6dc49921cf6"
server_key_iv: "17f7477d0a4100514335ac9f316c06d0"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.413621 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "f8babf411daa0381186ac6dc49921cf6"
server_key_iv: "17f7477d0a4100514335ac9f316c06d0"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.415122 11726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:24:54.435973 11743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.436271 11726 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260808 02:24:54.436434 11726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "f8babf411daa0381186ac6dc49921cf6"
server_key_iv: "17f7477d0a4100514335ac9f316c06d0"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.436614 11726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:54.456763 11726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:54.459087 11726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:54.459263 11726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:54.459548 11726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:54.459949 11726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:54.460014 11726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.460058 11726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:54.460085 11726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.467206 11726 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:44945
I20260808 02:24:54.467666 11726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:54.469133 11872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:44945 every 8 connection(s)
I20260808 02:24:54.472088 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 11726
I20260808 02:24:54.472211 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:54.472404 31360 external_mini_cluster.cc:1428] Setting key d290956b378029ab3240ecf663b836dc
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:54.480105 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.468605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40415 (local address 127.30.160.1:47361)
0808 02:24:54.469408 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:54.469455 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:54.470115 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:54.470473 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:54.470478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:54.470743 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:54.470993 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.471001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.472074 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.472075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.473410 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.473414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.474016 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.474017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.474161 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.474775 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:54.474791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:54.476096 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:54.478293 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.478299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.478302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.478592 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.478869 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.478870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.478871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.479010 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.479323 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:54.479330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:54.479487 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 02:24:54.479609 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 02:24:54.479764 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":384,"threads_started":1}
I20260808 02:24:54.480715 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.468609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:47361 (local address 127.30.160.62:40415)
0808 02:24:54.469338 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:54.469340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:54.469565 (+ 225us) server_negotiation.cc:407] Connection header received
0808 02:24:54.470252 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:54.470255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:54.470271 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:54.470377 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:54.471109 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.471953 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.473514 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.473854 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.473923 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.476192 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:54.476193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:54.476193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:54.476201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:54.478201 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.478689 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.478690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.478691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.478771 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.479089 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.479090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.479091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.479229 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:54.479293 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:54.480486 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0808 02:24:54.480586 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":92,"threads_started":1}
I20260808 02:24:54.481534 11873 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40415
I20260808 02:24:54.481686 11873 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:54.482017 11873 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:54.482666 11639 ts_manager.cc:194] Registered new tserver with Master: cbfa01227d9d46ae9972ea2c30f5bfe6 (127.30.160.1:44945)
I20260808 02:24:54.483546 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:47361
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:24:54.504009 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40415
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:54.659098 11882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:54.659389 11882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:54.659428 11882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:54.661167 11882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:54.661239 11882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:54.661283 11882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:24:54.663523 11882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.160.20:57071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:54.663764 11882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:54.664131 11882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:24:54.671069 11892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.671150 11889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.671468 11882 server_base.cc:1034] running on GCE node
W20260808 02:24:54.671291 11890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.672041 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:54.672349 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:54.673524 11882 hybrid_clock.cc:648] HybridClock initialized: now 1786155894673494 us; error 46 us; skew 500 ppm
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:54.676807 11882 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:24:54.677353 11882 webserver.cc:460] Webserver started at http://127.30.160.2:40959/ using document root <none> and password file <none>
I20260808 02:24:54.677614 11882 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:54.677665 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:54.677790 11882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:54.701092 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "119478de83e24208bc48d714998995cd"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "b2f8deba19133d3681d332dae607b776"
server_key_iv: "4922dcd93f60c7adc7441c84da90df63"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.701586 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "119478de83e24208bc48d714998995cd"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "b2f8deba19133d3681d332dae607b776"
server_key_iv: "4922dcd93f60c7adc7441c84da90df63"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.713052 11882 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.001s
I20260808 02:24:54.728036 11899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.728241 11882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260808 02:24:54.728348 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "119478de83e24208bc48d714998995cd"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "b2f8deba19133d3681d332dae607b776"
server_key_iv: "4922dcd93f60c7adc7441c84da90df63"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.728448 11882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:54.744593 11882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:54.747066 11882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:54.747287 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:54.747562 11882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:54.747989 11882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:54.748086 11882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.748159 11882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:54.748241 11882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.754900 11882 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38359
I20260808 02:24:54.755379 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:54.761997 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 11882
I20260808 02:24:54.762109 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:54.762297 31360 external_mini_cluster.cc:1428] Setting key 98d2f4903339171cabf918f0cc2d9d5c
I20260808 02:24:54.758260 12020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38359 every 8 connection(s)
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:54.770713 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.759099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:34665 (local address 127.30.160.62:40415)
0808 02:24:54.759164 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:54.759166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:54.759482 (+ 316us) server_negotiation.cc:407] Connection header received
0808 02:24:54.760239 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:54.760240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:54.760257 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:54.760289 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:54.761157 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.761930 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.764115 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.764453 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.764494 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.767158 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:54.767160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:54.767160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:54.767167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:54.768988 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.769509 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.769511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.769511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.769584 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.769913 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.769914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.769915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.770064 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:54.770142 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:54.770507 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 02:24:54.770616 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 02:24:54.770925 12023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.758820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40415 (local address 127.30.160.2:34665)
0808 02:24:54.759354 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:54.759399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:54.760100 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:54.760477 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:54.760481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:54.760715 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:54.761033 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.761039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.762188 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.762189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.764012 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.764015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.764593 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.764594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.764732 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.765296 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:54.765308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:54.767048 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:54.769089 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.769096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.769098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.769404 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.769676 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.769677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.769677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.769821 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.770149 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:54.770155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:54.770330 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 02:24:54.770464 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 02:24:54.770636 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":82,"threads_started":1}
I20260808 02:24:54.771502 12021 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40415
I20260808 02:24:54.771598 12021 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:54.771811 12021 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:54.772315 11639 ts_manager.cc:194] Registered new tserver with Master: 119478de83e24208bc48d714998995cd (127.30.160.2:38359)
I20260808 02:24:54.772814 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:34665
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:24:54.800762 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40415
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:54.923687 12026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:54.924036 12026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:54.924122 12026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:54.925776 12026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:54.925882 12026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:54.925964 12026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:24:54.928351 12026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.160.20:57071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:40415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:54.928792 12026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:54.929100 12026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:54.935668 12032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.936072 12031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.937127 12026 server_base.cc:1034] running on GCE node
W20260808 02:24:54.941557 12034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.941841 12026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:54.942139 12026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:54.943303 12026 hybrid_clock.cc:648] HybridClock initialized: now 1786155894943286 us; error 41 us; skew 500 ppm
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:54.948114 12026 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:24:54.948642 12026 webserver.cc:460] Webserver started at http://127.30.160.3:37643/ using document root <none> and password file <none>
I20260808 02:24:54.948922 12026 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:54.948972 12026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:54.949092 12026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:24:54 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:54.967811 12026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "cc7bc0d7ca634ba395a267f642d4aa1e"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "28713d77b611d6ceca8931469a0666aa"
server_key_iv: "338ee0e81833489e4e582a5f7477ba2c"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.968327 12026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "cc7bc0d7ca634ba395a267f642d4aa1e"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "28713d77b611d6ceca8931469a0666aa"
server_key_iv: "338ee0e81833489e4e582a5f7477ba2c"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.973100 12026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:24:54.996024 12041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.996222 12026 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260808 02:24:54.996337 12026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "cc7bc0d7ca634ba395a267f642d4aa1e"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "28713d77b611d6ceca8931469a0666aa"
server_key_iv: "338ee0e81833489e4e582a5f7477ba2c"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:54.996454 12026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:55.032459 12026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:55.039449 12026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:55.039611 12026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:55.039868 12026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:55.040343 12026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:55.040395 12026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.040437 12026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:55.040464 12026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.047086 12026 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:43657
I20260808 02:24:55.047528 12026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:55.050990 12164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:43657 every 8 connection(s)
I20260808 02:24:55.051630 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12026
I20260808 02:24:55.051714 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:55.051914 31360 external_mini_cluster.cc:1428] Setting key 025b175d9c3bfce4e0a31b6cb02c4c80
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:55.062407 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.051290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:45133 (local address 127.30.160.62:40415)
0808 02:24:55.051385 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.051387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.052546 (+ 1159us) server_negotiation.cc:407] Connection header received
0808 02:24:55.053324 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.053325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.053337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.053368 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.054193 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.054901 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.056317 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.056571 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.056612 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.059186 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.059188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.059188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.059195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.060934 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.061404 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.061405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.061406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.061464 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.061714 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.061715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.061715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.061836 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.061877 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.062212 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.062299 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:24:55.063311 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.051289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40415 (local address 127.30.160.3:45133)
0808 02:24:55.052251 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:55.052300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:55.053193 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:55.053552 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:55.053556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:55.053817 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:55.054087 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.054093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.055013 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.055014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.056207 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.056210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.056713 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.056714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.056876 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.057497 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:55.057508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:55.059081 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:55.061030 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.061036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.061038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.061320 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.061530 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.061531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.061531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.061647 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.061921 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:55.061926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:55.062073 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 02:24:55.062638 (+ 565us) client_negotiation.cc:240] Negotiation successful
0808 02:24:55.062789 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":92,"threads_started":1}
I20260808 02:24:55.064006 12165 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40415
I20260808 02:24:55.064093 12165 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.064270 12165 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:55.064694 11639 ts_manager.cc:194] Registered new tserver with Master: cc7bc0d7ca634ba395a267f642d4aa1e (127.30.160.3:43657)
I20260808 02:24:55.065199 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:45133
I20260808 02:24:55.074194 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:24:55.074900 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 11726
I20260808 02:24:55.081139 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 11882
I20260808 02:24:55.095602 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12026
I20260808 02:24:55.107823 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 11606
I20260808 02:24:55.115005 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40415
--webserver_interface=127.30.160.62
--webserver_port=43217
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:40415
--ranger_config_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:55.198607 12168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:55.198832 12168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:55.198871 12168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:55.200125 12168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:55.200184 12168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:55.200215 12168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:55.200246 12168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:55.200275 12168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:55.201812 12168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:40415
--ranger_config_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:55.202066 12168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:55.202279 12168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:24:55.206998 12168 server_base.cc:1034] running on GCE node
W20260808 02:24:55.206964 12173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.207055 12177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.206954 12175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.207355 12168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:55.207742 12168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:55.208909 12168 hybrid_clock.cc:648] HybridClock initialized: now 1786155895208880 us; error 46 us; skew 500 ppm
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.211591 12168 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:24:55.212105 12168 webserver.cc:460] Webserver started at http://127.30.160.62:43217/ using document root <none> and password file <none>
I20260808 02:24:55.212404 12168 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:55.212481 12168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:55.259011 12168 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:24:55.259968 12185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.260181 12168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:24:55.260277 12168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "14eda6e9881b4d5f96c20c713df9ba83"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-zkpd"
server_key: "0a2deec03765c565f814419a59ba90e2"
server_key_iv: "c793b565c10f4b5a5ed3e18ffbeacbfa"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:55.260756 12168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:55.293375 12168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:55.295516 12168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:55.295671 12168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:55.300935 12168 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:40415
I20260808 02:24:55.301067 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:40415 every 8 connection(s)
I20260808 02:24:55.301441 12168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:55.302753 12245 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:24:55.303429 12245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Bootstrap starting.
I20260808 02:24:55.311352 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12168
I20260808 02:24:55.311685 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:44945
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:40415
--webserver_port=33497
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:24:55.311745 12245 log.cc:824] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:55.313912 12245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:24:55.314218 12245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Bootstrap complete.
I20260808 02:24:55.316779 12245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:55.317019 12245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eda6e9881b4d5f96c20c713df9ba83, State: Initialized, Role: FOLLOWER
I20260808 02:24:55.317183 12245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:55.317255 12245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:55.317312 12245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:55.317363 12245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:24:55.318354 12245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:55.318444 12245 leader_election.cc:302] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14eda6e9881b4d5f96c20c713df9ba83; no voters:
I20260808 02:24:55.318643 12245 leader_election.cc:288] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:24:55.318682 12249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:24:55.318796 12249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [term 2 LEADER]: Becoming Leader. State: Replica: 14eda6e9881b4d5f96c20c713df9ba83, State: Running, Role: LEADER
I20260808 02:24:55.318894 12249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } }
I20260808 02:24:55.318935 12245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:55.319159 12249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "14eda6e9881b4d5f96c20c713df9ba83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } } }
I20260808 02:24:55.319228 12249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:55.319353 12249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14eda6e9881b4d5f96c20c713df9ba83. Latest consensus state: current_term: 2 leader_uuid: "14eda6e9881b4d5f96c20c713df9ba83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14eda6e9881b4d5f96c20c713df9ba83" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 40415 } } }
I20260808 02:24:55.319509 12249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:55.319873 12253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:55.320480 12253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:55.320659 12253 catalog_manager.cc:1259] Loaded cluster ID: a200ba4f34fc4800a5ca96bd9718ebed
I20260808 02:24:55.320735 12253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:55.320856 12245 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:24:55.321993 12253 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:55.322180 12253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Loaded TSK: 0
I20260808 02:24:55.322436 12253 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:24:55.426761 12248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:55.427116 12248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:55.427227 12248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:55.428906 12248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:55.428969 12248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:55.429006 12248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:24:55.431006 12248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:44945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=33497
--tserver_master_addrs=127.30.160.62:40415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:55.431450 12248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:55.431815 12248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:55.438274 12284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.438346 12285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.438630 12248 server_base.cc:1034] running on GCE node
W20260808 02:24:55.438886 12287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.439145 12248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:55.439430 12248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:55.440642 12248 hybrid_clock.cc:648] HybridClock initialized: now 1786155895440619 us; error 37 us; skew 500 ppm
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.444085 12248 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:24:55.444619 12248 webserver.cc:460] Webserver started at http://127.30.160.1:33497/ using document root <none> and password file <none>
I20260808 02:24:55.444949 12248 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:55.445093 12248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:55.488346 12248 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260808 02:24:55.496004 12295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.496255 12248 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260808 02:24:55.496367 12248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "f8babf411daa0381186ac6dc49921cf6"
server_key_iv: "17f7477d0a4100514335ac9f316c06d0"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:55.496860 12248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:55.513484 12248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:55.516076 12248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:55.516283 12248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:55.516564 12248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:55.516986 12248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:55.517086 12248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.517208 12248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:55.517288 12248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.525211 12248 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:44945
I20260808 02:24:55.525708 12248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:55.526026 12413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:44945 every 8 connection(s)
I20260808 02:24:55.534534 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12248
I20260808 02:24:55.534830 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:38359
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:40415
--webserver_port=40959
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:55.542568 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.526438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40415 (local address 127.30.160.1:47057)
0808 02:24:55.526634 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:55.526683 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:55.527376 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:55.527973 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:55.527978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:55.528205 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:55.528492 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.528499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.532484 (+ 3985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.532486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.533771 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.533774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.536499 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.536501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.536637 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.537187 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:55.537195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:55.538858 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:55.540934 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.540941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.540943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.541161 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.541436 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.541437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.541438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.541550 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.541938 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:55.541945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:55.542078 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 02:24:55.542184 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:24:55.542345 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260808 02:24:55.542555 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.526618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:47057 (local address 127.30.160.62:40415)
0808 02:24:55.526837 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.526850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.526892 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:24:55.527626 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.527633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.527693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.527875 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.528658 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.532350 (+ 3692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.533879 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.536368 (+ 2489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.536505 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.538982 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.538986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.538987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.538999 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.540826 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.541252 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.541255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.541256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.541360 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.541624 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.541625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.541625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.541858 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.541942 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.542247 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.542358 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260808 02:24:55.543696 12414 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40415
I20260808 02:24:55.543823 12414 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.544030 12414 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
W20260808 02:24:55.544911 12414 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:40415 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:24:55.544996 12414 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.545070 12414 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:55.545392 12414 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.545466 12414 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
W20260808 02:24:55.545751 12414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:24:55.648802 12418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:55.649021 12418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:55.649060 12418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:55.650301 12418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:55.650358 12418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:55.650403 12418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:24:55.652478 12418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:38359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=40959
--tserver_master_addrs=127.30.160.62:40415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:55.652712 12418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:55.653026 12418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:55.659162 12426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.659174 12424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.659379 12423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.659812 12418 server_base.cc:1034] running on GCE node
I20260808 02:24:55.660071 12418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:55.660341 12418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:55.661527 12418 hybrid_clock.cc:648] HybridClock initialized: now 1786155895661504 us; error 36 us; skew 500 ppm
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.664773 12418 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:24:55.665346 12418 webserver.cc:460] Webserver started at http://127.30.160.2:40959/ using document root <none> and password file <none>
I20260808 02:24:55.665608 12418 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:55.665700 12418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:55.688603 12418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:24:55.689714 12434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.689894 12418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:24:55.689983 12418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "119478de83e24208bc48d714998995cd"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "b2f8deba19133d3681d332dae607b776"
server_key_iv: "4922dcd93f60c7adc7441c84da90df63"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:55.690445 12418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:55.727172 12418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:55.729745 12418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:55.729929 12418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:55.730294 12418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:55.730767 12418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:55.730887 12418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.730976 12418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:55.731056 12418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.737917 12418 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38359
I20260808 02:24:55.738621 12418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:55.741216 12557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38359 every 8 connection(s)
I20260808 02:24:55.743337 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12418
I20260808 02:24:55.743649 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.160.20:57071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:43657
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:40415
--webserver_port=37643
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:55.756769 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.746012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:50549 (local address 127.30.160.62:40415)
0808 02:24:55.746064 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.746066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.746335 (+ 269us) server_negotiation.cc:407] Connection header received
0808 02:24:55.746991 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.746992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.747008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.747087 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.747961 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.748758 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.750305 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.750621 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.750701 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.753367 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.753369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.753370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.753377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.755100 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.755615 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.755616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.755617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.755690 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.755996 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.755997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.755998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.756131 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.756176 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.756589 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.756672 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:24:55.757507 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.745662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40415 (local address 127.30.160.2:50549)
0808 02:24:55.746173 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:55.746219 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:55.746851 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:55.747235 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:55.747240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:55.747503 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:55.747803 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.747811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.748886 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.748886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.750193 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.750196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.750749 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.750751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.750895 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.751546 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:55.751557 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:55.753236 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:55.755200 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.755206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.755209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.755517 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.755764 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.755765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.755765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.755893 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.756226 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:55.756232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:55.756422 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 02:24:55.757096 (+ 674us) client_negotiation.cc:240] Negotiation successful
0808 02:24:55.757268 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":89,"threads_started":1}
I20260808 02:24:55.758260 12558 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40415
I20260808 02:24:55.758397 12558 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.758616 12558 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
W20260808 02:24:55.759370 12558 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:40415 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:24:55.759454 12558 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.759536 12558 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:55.760066 12558 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.760161 12558 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
W20260808 02:24:55.760526 12558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:24:55.867440 12560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:55.867637 12560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:55.867660 12560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:55.868858 12560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:55.868901 12560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:55.868927 12560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:24:55.870561 12560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.160.20:57071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:43657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=37643
--tserver_master_addrs=127.30.160.62:40415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:55.870786 12560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:55.871060 12560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:55.877058 12569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.877053 12572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.877081 12568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.877488 12560 server_base.cc:1034] running on GCE node
I20260808 02:24:55.877698 12560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:55.878010 12560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:55.881350 12560 hybrid_clock.cc:648] HybridClock initialized: now 1786155895879890 us; error 1472 us; skew 500 ppm
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.884644 12560 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:24:55.885219 12560 webserver.cc:460] Webserver started at http://127.30.160.3:37643/ using document root <none> and password file <none>
I20260808 02:24:55.885576 12560 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:55.885704 12560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:55.900321 12560 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:24:55.901423 12581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.901618 12560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:24:55.901705 12560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "cc7bc0d7ca634ba395a267f642d4aa1e"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-zkpd"
server_key: "28713d77b611d6ceca8931469a0666aa"
server_key_iv: "338ee0e81833489e4e582a5f7477ba2c"
server_key_version: "kuduclusterkey@0"
I20260808 02:24:55.902197 12560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:55.916105 12560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:55.918399 12560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:55.918560 12560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:55.918828 12560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:55.919188 12560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:55.919242 12560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.919281 12560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:55.919332 12560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.926667 12560 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:43657
I20260808 02:24:55.926752 12700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:43657 every 8 connection(s)
I20260808 02:24:55.927131 12560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:55.930755 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12560
Aug 08 02:24:55 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:55.938050 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.928018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:54103 (local address 127.30.160.62:40415)
0808 02:24:55.928068 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.928071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.928345 (+ 274us) server_negotiation.cc:407] Connection header received
0808 02:24:55.928907 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.928909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.928921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.929006 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.929611 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.930112 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.931460 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.931750 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.931829 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.934618 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.934620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.934620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.934628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.936359 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.936901 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.936902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.936902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.936962 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.937267 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.937268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.937268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.937405 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.937449 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.937844 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.937933 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:24:55.938648 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.928013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:40415 (local address 127.30.160.3:54103)
0808 02:24:55.928180 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:55.928228 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:55.928748 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:55.929058 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:55.929062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:55.929241 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:55.929504 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.929510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.930210 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.930211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.931372 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.931374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.931853 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.931854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.932005 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.932472 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:55.932480 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:55.934512 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:55.936436 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.936443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.936446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.936748 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.937042 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.937043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.937044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.937179 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.937498 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:55.937504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:55.937665 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 02:24:55.938186 (+ 521us) client_negotiation.cc:240] Negotiation successful
0808 02:24:55.938333 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260808 02:24:55.939211 12701 heartbeater.cc:347] Connected to a master server at 127.30.160.62:40415
I20260808 02:24:55.939316 12701 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.939496 12701 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
W20260808 02:24:55.940122 12701 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:40415 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:24:55.940218 12701 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.940301 12701 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:55.940631 12701 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.940707 12701 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
W20260808 02:24:55.940970 12701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:24:55.941448 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.931150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.30.160.62:40415)
0808 02:24:55.931315 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.931318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.931469 (+ 151us) server_negotiation.cc:407] Connection header received
0808 02:24:55.931569 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.931569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.931581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.931646 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.932849 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.933384 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.934255 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.934578 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.934635 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.936462 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.936462 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.936463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.936469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.938932 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.939701 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.939702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.939703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.939765 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.940428 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.940429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.940430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.940568 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.941138 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.941196 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.941383 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
Aug 08 02:24:56 dist-test-slave-zkpd krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155896, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:56.545958 12414 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:56.546118 12414 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:56.546536 12205 ts_manager.cc:194] Registered new tserver with Master: cbfa01227d9d46ae9972ea2c30f5bfe6 (127.30.160.1:44945)
I20260808 02:24:56.547581 12205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:47057
I20260808 02:24:56.760721 12558 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:56.760881 12558 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:56.762297 12205 ts_manager.cc:194] Registered new tserver with Master: 119478de83e24208bc48d714998995cd (127.30.160.2:38359)
I20260808 02:24:56.762874 12205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:50549
Aug 08 02:24:56 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155896, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
I20260808 02:24:56.941151 12701 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:56.941321 12701 heartbeater.cc:511] Master 127.30.160.62:40415 requested a full tablet report, sending...
I20260808 02:24:56.941641 12204 ts_manager.cc:194] Registered new tserver with Master: cc7bc0d7ca634ba395a267f642d4aa1e (127.30.160.3:43657)
I20260808 02:24:56.942200 12204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:54103
I20260808 02:24:56.947934 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:24:56.956841 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.948944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.30.160.62:40415)
0808 02:24:56.949084 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:56.949086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:56.949243 (+ 157us) server_negotiation.cc:407] Connection header received
0808 02:24:56.949281 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:56.949282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:56.949295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:56.949388 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:56.950119 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.950883 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.951963 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.952253 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.952325 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.953826 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:56.953827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:56.953828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:56.953835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:56.955563 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:56.956003 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:56.956004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:56.956005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:56.956072 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:56.956284 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:56.956285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:56.956285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:56.956421 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:56.956517 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:56.956658 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 02:24:56.956742 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260808 02:24:56.959038 12204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:56.961346 12204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:56.972820 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.964624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43657 (local address 127.0.0.1:48186)
0808 02:24:56.968223 (+ 3599us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:56.968232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:56.968267 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:56.968842 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:56.968845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:56.968847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:56.968919 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:56.968921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.970278 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.970279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:56.971529 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:56.971532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.972531 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.972532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:56.972606 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.972621 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:24:56.972667 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 02:24:56.972725 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3586,"thread_start_us":95,"threads_started":1}
I20260808 02:24:56.973385 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.964977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.30.160.3:43657)
0808 02:24:56.968651 (+ 3674us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:56.968655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:56.968665 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:24:56.968698 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:56.968700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:56.968715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:56.969000 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:56.969090 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.970186 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.971642 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.972430 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.973174 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.973242 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:56.973270 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 02:24:56.973306 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3641,"thread_start_us":67,"threads_started":1}
I20260808 02:24:56.973912 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.964360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:44945 (local address 127.0.0.1:48072)
0808 02:24:56.967994 (+ 3634us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:56.968044 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:56.968116 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:56.970070 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:56.970071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:56.970073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:56.970121 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:56.970122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.971327 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.971328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:56.972416 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:56.972419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.973715 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.973716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:56.973776 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.973780 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:56.973809 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 02:24:56.973845 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3617,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260808 02:24:56.974272 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.964994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.30.160.1:44945)
0808 02:24:56.965447 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:56.965453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:56.969917 (+ 4464us) server_negotiation.cc:407] Connection header received
0808 02:24:56.969959 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:56.969961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:56.969977 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:56.970181 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:56.970253 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.971221 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.972739 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.973607 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.974074 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.974140 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:56.974165 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 02:24:56.974202 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":87,"threads_started":1}
I20260808 02:24:56.974251 12632 tablet_service.cc:1467] Processing CreateTablet for tablet 3f491ba70aa2410faa4004d02323a075 (DEFAULT_TABLE table=test-table [id=808bff211e664eb19c1b8c28d8df7dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:56.974650 12632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f491ba70aa2410faa4004d02323a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:56.974927 12339 tablet_service.cc:1467] Processing CreateTablet for tablet 3f491ba70aa2410faa4004d02323a075 (DEFAULT_TABLE table=test-table [id=808bff211e664eb19c1b8c28d8df7dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:56.975250 12339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f491ba70aa2410faa4004d02323a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:56.978336 12736 tablet_bootstrap.cc:492] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Bootstrap starting.
I20260808 02:24:56.978771 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.964359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38359 (local address 127.0.0.1:58128)
0808 02:24:56.967994 (+ 3635us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:56.968044 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:56.968117 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:56.969872 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:56.969873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:56.969876 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:56.969917 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:56.969918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.971150 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.971150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:56.972650 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:56.972652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.973652 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.973653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:56.973733 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.973737 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:56.978636 (+ 4899us) client_negotiation.cc:240] Negotiation successful
0808 02:24:56.978685 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3595,"mutex_wait_us":431,"thread_start_us":102,"threads_started":1}
I20260808 02:24:56.979081 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:56.964642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.30.160.2:38359)
0808 02:24:56.968372 (+ 3730us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:56.968377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:56.968388 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:24:56.968425 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:56.968427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:56.968443 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:56.968515 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:56.970010 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.971054 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.972734 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:56.973500 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:56.973564 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:56.973612 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:56.978946 (+ 5334us) server_negotiation.cc:299] Negotiation successful
0808 02:24:56.978994 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3695,"thread_start_us":68,"threads_started":1}
I20260808 02:24:56.979105 12736 tablet_bootstrap.cc:654] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:56.979418 12736 log.cc:824] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:56.979713 12480 tablet_service.cc:1467] Processing CreateTablet for tablet 3f491ba70aa2410faa4004d02323a075 (DEFAULT_TABLE table=test-table [id=808bff211e664eb19c1b8c28d8df7dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:56.980091 12480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f491ba70aa2410faa4004d02323a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:56.980276 12736 tablet_bootstrap.cc:492] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: No bootstrap required, opened a new log
I20260808 02:24:56.980343 12736 ts_tablet_manager.cc:1397] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:24:56.982652 12736 raft_consensus.cc:348] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:56.982803 12736 raft_consensus.cc:374] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:56.982836 12736 raft_consensus.cc:729] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7bc0d7ca634ba395a267f642d4aa1e, State: Initialized, Role: FOLLOWER
I20260808 02:24:56.982942 12736 consensus_queue.cc:260] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:56.983186 12736 ts_tablet_manager.cc:1428] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:56.983667 12701 heartbeater.cc:503] Master 127.30.160.62:40415 was elected leader, sending a full tablet report...
I20260808 02:24:56.984835 12740 tablet_bootstrap.cc:492] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Bootstrap starting.
I20260808 02:24:56.985515 12740 tablet_bootstrap.cc:654] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:56.985904 12740 log.cc:824] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:56.987330 12737 tablet_bootstrap.cc:492] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: Bootstrap starting.
I20260808 02:24:56.988041 12737 tablet_bootstrap.cc:654] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:56.988431 12737 log.cc:824] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:56.989009 12740 tablet_bootstrap.cc:492] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: No bootstrap required, opened a new log
I20260808 02:24:56.989112 12740 ts_tablet_manager.cc:1397] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 02:24:56.989264 12737 tablet_bootstrap.cc:492] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: No bootstrap required, opened a new log
I20260808 02:24:56.989344 12737 ts_tablet_manager.cc:1397] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:24:56.991518 12740 raft_consensus.cc:348] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:56.991669 12740 raft_consensus.cc:374] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:56.991551 12737 raft_consensus.cc:348] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:56.991703 12740 raft_consensus.cc:729] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119478de83e24208bc48d714998995cd, State: Initialized, Role: FOLLOWER
I20260808 02:24:56.991708 12737 raft_consensus.cc:374] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:56.991747 12737 raft_consensus.cc:729] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfa01227d9d46ae9972ea2c30f5bfe6, State: Initialized, Role: FOLLOWER
I20260808 02:24:56.991837 12740 consensus_queue.cc:260] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:56.991858 12737 consensus_queue.cc:260] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:56.992149 12740 ts_tablet_manager.cc:1428] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:56.992151 12737 ts_tablet_manager.cc:1428] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 02:24:56.992262 12558 heartbeater.cc:503] Master 127.30.160.62:40415 was elected leader, sending a full tablet report...
I20260808 02:24:56.992465 12414 heartbeater.cc:503] Master 127.30.160.62:40415 was elected leader, sending a full tablet report...
W20260808 02:24:56.996879 12559 tablet.cc:2369] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:57.005818 12745 raft_consensus.cc:482] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:57.005936 12745 raft_consensus.cc:504] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:57.006816 12745 leader_election.cc:288] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 119478de83e24208bc48d714998995cd (127.30.160.2:38359), cc7bc0d7ca634ba395a267f642d4aa1e (127.30.160.3:43657)
I20260808 02:24:57.010519 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.006730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:48297 (local address 127.30.160.3:43657)
0808 02:24:57.006779 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:57.006782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:57.007044 (+ 262us) server_negotiation.cc:407] Connection header received
0808 02:24:57.007109 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:57.007111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:57.007122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:57.007182 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:57.007430 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.008147 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.009556 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.010101 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.010155 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:57.010165 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:57.010409 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:24:57.010451 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:24:57.011042 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.006456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43657 (local address 127.30.160.1:48297)
0808 02:24:57.006974 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:57.006986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:57.007053 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:57.007239 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:57.007240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:57.007242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:57.007341 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:57.007343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.008252 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.008253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:57.009442 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:57.009445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.010241 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.010242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:57.010319 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:57.010323 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:57.010901 (+ 578us) client_negotiation.cc:240] Negotiation successful
0808 02:24:57.010947 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":141,"threads_started":1}
I20260808 02:24:57.011564 12643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f491ba70aa2410faa4004d02323a075" candidate_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" is_pre_election: true
I20260808 02:24:57.011734 12643 raft_consensus.cc:2454] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbfa01227d9d46ae9972ea2c30f5bfe6 in term 0.
I20260808 02:24:57.012050 12296 leader_election.cc:302] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbfa01227d9d46ae9972ea2c30f5bfe6, cc7bc0d7ca634ba395a267f642d4aa1e; no voters:
I20260808 02:24:57.012182 12745 raft_consensus.cc:2781] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:57.012225 12745 raft_consensus.cc:482] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:57.012246 12745 raft_consensus.cc:3037] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:57.013230 12745 raft_consensus.cc:504] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:57.013367 12745 leader_election.cc:288] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [CANDIDATE]: Term 1 election: Requested vote from peers 119478de83e24208bc48d714998995cd (127.30.160.2:38359), cc7bc0d7ca634ba395a267f642d4aa1e (127.30.160.3:43657)
I20260808 02:24:57.013674 12643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f491ba70aa2410faa4004d02323a075" candidate_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e"
I20260808 02:24:57.013772 12643 raft_consensus.cc:3037] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:57.013818 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.006836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:50095 (local address 127.30.160.2:38359)
0808 02:24:57.006874 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:57.006876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:57.007185 (+ 309us) server_negotiation.cc:407] Connection header received
0808 02:24:57.007274 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:57.007275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:57.007283 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:57.007309 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:57.007513 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.008225 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.012742 (+ 4517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.013309 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.013383 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:57.013394 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:57.013654 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 02:24:57.013729 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:24:57.014074 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.006464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38359 (local address 127.30.160.1:50095)
0808 02:24:57.007139 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:57.007147 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:57.007208 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:57.007380 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:57.007381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:57.007382 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:57.007447 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:57.007449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.011506 (+ 4057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.011507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:57.012632 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:57.012635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.013470 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.013471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:57.013559 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:57.013564 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:24:57.013918 (+ 354us) client_negotiation.cc:240] Negotiation successful
0808 02:24:57.013973 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":140,"threads_started":1}
I20260808 02:24:57.014446 12503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f491ba70aa2410faa4004d02323a075" candidate_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119478de83e24208bc48d714998995cd" is_pre_election: true
I20260808 02:24:57.014616 12503 raft_consensus.cc:2454] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbfa01227d9d46ae9972ea2c30f5bfe6 in term 0.
I20260808 02:24:57.014669 12643 raft_consensus.cc:2454] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbfa01227d9d46ae9972ea2c30f5bfe6 in term 1.
I20260808 02:24:57.014699 12505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f491ba70aa2410faa4004d02323a075" candidate_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119478de83e24208bc48d714998995cd"
I20260808 02:24:57.014761 12505 raft_consensus.cc:3037] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:57.014854 12296 leader_election.cc:302] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbfa01227d9d46ae9972ea2c30f5bfe6, cc7bc0d7ca634ba395a267f642d4aa1e; no voters:
I20260808 02:24:57.014933 12745 raft_consensus.cc:2781] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:57.015077 12745 raft_consensus.cc:686] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [term 1 LEADER]: Becoming Leader. State: Replica: cbfa01227d9d46ae9972ea2c30f5bfe6, State: Running, Role: LEADER
I20260808 02:24:57.015167 12745 consensus_queue.cc:237] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } }
I20260808 02:24:57.015635 12505 raft_consensus.cc:2454] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbfa01227d9d46ae9972ea2c30f5bfe6 in term 1.
I20260808 02:24:57.016031 12205 catalog_manager.cc:5495] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbfa01227d9d46ae9972ea2c30f5bfe6 (127.30.160.1). New cstate: current_term: 1 leader_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbfa01227d9d46ae9972ea2c30f5bfe6" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 } health_report { overall_health: UNKNOWN } } }
W20260808 02:24:57.027091 12415 tablet.cc:2369] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:57.030499 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.027336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.30.160.1:44945)
0808 02:24:57.027388 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:57.027390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:57.027400 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:24:57.027430 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:57.027431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:57.027449 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:57.027519 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:57.027677 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.028406 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.029458 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.029730 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.029796 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:57.030001 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:57.030136 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:57.030210 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:57.030392 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:24:57.030428 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 02:24:57.032991 12643 raft_consensus.cc:1261] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 1 FOLLOWER]: Refusing update from remote peer cbfa01227d9d46ae9972ea2c30f5bfe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:57.033059 12505 raft_consensus.cc:1261] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 1 FOLLOWER]: Refusing update from remote peer cbfa01227d9d46ae9972ea2c30f5bfe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:57.033411 12748 consensus_queue.cc:1035] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "119478de83e24208bc48d714998995cd" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:57.033610 12745 consensus_queue.cc:1035] T 3f491ba70aa2410faa4004d02323a075 P cbfa01227d9d46ae9972ea2c30f5bfe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7bc0d7ca634ba395a267f642d4aa1e" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:57.037899 12751 mvcc.cc:205] Tried to move back new op lower bound from 7316094554244050944 to 7316094554174746624. Current Snapshot: MvccSnapshot[applied={T|T < 7316094554244050944}]
I20260808 02:24:57.038054 12754 mvcc.cc:205] Tried to move back new op lower bound from 7316094554244050944 to 7316094554174746624. Current Snapshot: MvccSnapshot[applied={T|T < 7316094554244050944}]
I20260808 02:24:57.041117 12205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:57.041218 12205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:57.041987 12205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14eda6e9881b4d5f96c20c713df9ba83: Sending DeleteTablet for 3 replicas of tablet 3f491ba70aa2410faa4004d02323a075
I20260808 02:24:57.042608 12480 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f491ba70aa2410faa4004d02323a075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:57 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:58128
I20260808 02:24:57.042853 12632 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f491ba70aa2410faa4004d02323a075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:57 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:48186
I20260808 02:24:57.043195 12752 mvcc.cc:205] Tried to move back new op lower bound from 7316094554244050944 to 7316094554174746624. Current Snapshot: MvccSnapshot[applied={T|T < 7316094554244050944}]
I20260808 02:24:57.043555 12339 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f491ba70aa2410faa4004d02323a075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:57 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:48072
I20260808 02:24:57.043814 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12248
I20260808 02:24:57.049618 12763 tablet_replica.cc:330] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: stopping tablet replica
I20260808 02:24:57.049782 12764 tablet_replica.cc:330] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: stopping tablet replica
I20260808 02:24:57.052004 12763 raft_consensus.cc:2229] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:57.052201 12764 raft_consensus.cc:2229] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:57.053346 12763 raft_consensus.cc:2258] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:57.052392 12764 raft_consensus.cc:2258] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:57.054209 12764 ts_tablet_manager.cc:1905] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:57.054895 12763 ts_tablet_manager.cc:1905] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:57.056886 12763 ts_tablet_manager.cc:1918] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:57.057334 12763 log.cc:1196] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/3f491ba70aa2410faa4004d02323a075
I20260808 02:24:57.057716 12763 ts_tablet_manager.cc:1939] T 3f491ba70aa2410faa4004d02323a075 P 119478de83e24208bc48d714998995cd: Deleting consensus metadata
I20260808 02:24:57.058221 12190 catalog_manager.cc:4841] TS 119478de83e24208bc48d714998995cd (127.30.160.2:38359): tablet 3f491ba70aa2410faa4004d02323a075 (table test-table [id=808bff211e664eb19c1b8c28d8df7dfc]) successfully deleted
I20260808 02:24:57.059804 12764 ts_tablet_manager.cc:1918] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:57.059880 12764 log.cc:1196] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/3f491ba70aa2410faa4004d02323a075
I20260808 02:24:57.060184 12764 ts_tablet_manager.cc:1939] T 3f491ba70aa2410faa4004d02323a075 P cc7bc0d7ca634ba395a267f642d4aa1e: Deleting consensus metadata
I20260808 02:24:57.060910 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12418
Aug 08 02:24:57 dist-test-slave-zkpd krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155896, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for HTTP/127.30.160.20@KRBTEST.COM
W20260808 02:24:57.061473 12188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:44945 (error 108)
I20260808 02:24:57.062342 12186 catalog_manager.cc:4841] TS cc7bc0d7ca634ba395a267f642d4aa1e (127.30.160.3:43657): tablet 3f491ba70aa2410faa4004d02323a075 (table test-table [id=808bff211e664eb19c1b8c28d8df7dfc]) successfully deleted
I20260808 02:24:57.062840 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.062708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:44945 (local address 127.0.0.1:48090)
0808 02:24:57.062751 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:57.062792 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260808 02:24:57.063061 12188 catalog_manager.cc:4584] TS cbfa01227d9d46ae9972ea2c30f5bfe6 (127.30.160.1:44945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f491ba70aa2410faa4004d02323a075: Network error: Client connection negotiation failed: client connection to 127.30.160.1:44945: connect: Connection refused (error 111)
I20260808 02:24:57.067678 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12560
I20260808 02:24:57.079547 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12168
I20260808 02:24:57.089272 31360 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:24:57.443320 31360 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:24:57.443411 31360 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:24:57.787925 31360 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:24:57.788 UTC [11157] LOG: received smart shutdown request
2026-08-08 02:24:57.792 UTC [11157] LOG: background worker "logical replication launcher" (PID 11167) exited with exit code 1
2026-08-08 02:24:57.792 UTC [11161] LOG: shutting down
2026-08-08 02:24:57.944 UTC [11157] LOG: database system is shut down
2026-08-08T02:24:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48317 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:58 dist-test-slave-zkpd krb5kdc[12777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:58 dist-test-slave-zkpd krb5kdc[12777](info): set up 2 sockets
Aug 08 02:24:58 dist-test-slave-zkpd krb5kdc[12777](info): commencing operation
krb5kdc: starting...
W20260808 02:24:59.219786 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:24:59.339382 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:42409
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:42409,127.30.160.61:45205,127.30.160.60:38353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:59.421748 12791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:59.421988 12791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:59.422027 12791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:59.423161 12791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:59.423240 12791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:59.423269 12791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:59.423300 12791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:59.423323 12791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:59.424892 12791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:42409,127.30.160.61:45205,127.30.160.60:38353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:42409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:59.425168 12791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:59.425408 12791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:59.430277 12797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.430274 12796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.430307 12799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:59.430593 12791 server_base.cc:1034] running on GCE node
I20260808 02:24:59.430830 12791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:59.431067 12791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:59.432261 12791 hybrid_clock.cc:648] HybridClock initialized: now 1786155899432242 us; error 30 us; skew 500 ppm
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:59.434736 12791 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:24:59.435209 12791 webserver.cc:460] Webserver started at http://127.30.160.62:33617/ using document root <none> and password file <none>
I20260808 02:24:59.435467 12791 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:59.435513 12791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:59.435680 12791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:59.436816 12791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "1809e9ab0d7c44ceba887238003e3c03"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "8cda972aa4c4a85c8d684d8701d8dd6d"
server_key_iv: "200bbdc75744f5c41829b51c167607fa"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.437204 12791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "1809e9ab0d7c44ceba887238003e3c03"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "8cda972aa4c4a85c8d684d8701d8dd6d"
server_key_iv: "200bbdc75744f5c41829b51c167607fa"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.438503 12791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:24:59.439200 12806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.439378 12791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:24:59.439479 12791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "1809e9ab0d7c44ceba887238003e3c03"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "8cda972aa4c4a85c8d684d8701d8dd6d"
server_key_iv: "200bbdc75744f5c41829b51c167607fa"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.439565 12791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:59.448573 12791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:59.450363 12791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:59.450527 12791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:59.454732 12791 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:42409
I20260808 02:24:59.454761 12861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:42409 every 8 connection(s)
I20260808 02:24:59.455096 12791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:59.456020 12862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:59.457451 12862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.464288 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12791
I20260808 02:24:59.464422 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:24:59.464633 31360 external_mini_cluster.cc:1428] Setting key a6f0bd008eee8276a74267ad2bf2f747
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:59.470424 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.458059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.30.160.62:42409)
0808 02:24:59.458305 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.458311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.458366 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:24:59.459116 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.459119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.459142 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.459252 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.460195 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.461629 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.463100 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.463565 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.463669 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.466450 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.466457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.466457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.466464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.468839 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.469310 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.469313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.469313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.469392 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.469651 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.469651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.469652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.469843 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.469923 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.470168 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.470258 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260808 02:24:59.470534 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.458014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:42409 (local address 127.0.0.1:54722)
0808 02:24:59.458181 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.458222 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.458935 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.459376 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.459379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.459803 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.460068 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.460074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.461756 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.461757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.463002 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.463005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.463774 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.463775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.463849 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.464513 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.464525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.466351 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.468935 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.468943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.468946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.469220 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.469466 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.469467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.469467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.469581 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.469937 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.469944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.470025 (+ 81us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.470133 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.470343 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":41,"thread_start_us":86,"threads_started":1}
I20260808 02:24:59.471283 12862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.471681 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.471534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44170)
0808 02:24:59.471599 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.471651 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 02:24:59.471812 12807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
I20260808 02:24:59.472194 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.472129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44184)
0808 02:24:59.472154 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.472178 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260808 02:24:59.472317 12862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:45205: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.30.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab.
Entry for principal kudu/127.30.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab.
I20260808 02:24:59.491040 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:45205
--webserver_interface=127.30.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab
--principal=kudu/127.30.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:42409,127.30.160.61:45205,127.30.160.60:38353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:24:59.521553 12862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } attempt: 1
I20260808 02:24:59.522040 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.521897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44200)
0808 02:24:59.521970 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.522008 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 02:24:59.522387 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.522286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44204)
0808 02:24:59.522332 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.522357 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 02:24:59.522475 12862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:45205: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
W20260808 02:24:59.572947 12870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:59.573148 12870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:59.573169 12870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:59.574321 12870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:59.574361 12870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:59.574375 12870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:59.574388 12870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:59.574400 12870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:59.575930 12870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:42409,127.30.160.61:45205,127.30.160.60:38353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab
--principal=kudu/127.30.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:59.576190 12870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:59.576468 12870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:59.582551 12878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.582590 12875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.582732 12876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:59.582787 12870 server_base.cc:1034] running on GCE node
I20260808 02:24:59.583051 12870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:59.583376 12870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:59.584798 12870 hybrid_clock.cc:648] HybridClock initialized: now 1786155899584761 us; error 55 us; skew 500 ppm
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:59.588184 12870 init.cc:377] Logged in from keytab as kudu/127.30.160.61@KRBTEST.COM (short username kudu)
I20260808 02:24:59.588685 12870 webserver.cc:460] Webserver started at http://127.30.160.61:42629/ using document root <none> and password file <none>
I20260808 02:24:59.588972 12870 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:59.589046 12870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:59.589208 12870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:59.590364 12870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data/instance:
uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "078a63d3eb7a3d31c9c9597a0f262819"
server_key_iv: "d6420eb583361f70d4107e7ad5d52d20"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.590644 12862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } attempt: 2
I20260808 02:24:59.590811 12870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal/instance:
uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "078a63d3eb7a3d31c9c9597a0f262819"
server_key_iv: "d6420eb583361f70d4107e7ad5d52d20"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.591092 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.590935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44218)
0808 02:24:59.590994 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.591039 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 02:24:59.591491 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.591369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44220)
0808 02:24:59.591440 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.591468 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 02:24:59.591634 12862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:45205: Network error: Client connection negotiation failed: client connection to 127.30.160.61:45205: connect: Connection refused (error 111)
I20260808 02:24:59.592594 12870 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260808 02:24:59.593276 12885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.593478 12870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.593570 12870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal
uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "078a63d3eb7a3d31c9c9597a0f262819"
server_key_iv: "d6420eb583361f70d4107e7ad5d52d20"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.593645 12870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:59.607266 12870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:59.609030 12870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:59.609184 12870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:59.613247 12870 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.61:45205
I20260808 02:24:59.613277 12943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.61:45205 every 8 connection(s)
I20260808 02:24:59.613602 12870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
I20260808 02:24:59.614439 12944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:59.616074 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12870
I20260808 02:24:59.616192 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-1/wal/instance
I20260808 02:24:59.615967 12944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.616384 31360 external_mini_cluster.cc:1428] Setting key 2da049f9c150171be3e37350250c0233
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.160.60@KRBTEST.COM; defaulting to no policy
I20260808 02:24:59.625449 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.616820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.30.160.62:42409)
0808 02:24:59.616881 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.616884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.617051 (+ 167us) server_negotiation.cc:407] Connection header received
0808 02:24:59.617485 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.617487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.617499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.617574 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.618238 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.618717 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.619836 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.620086 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.620141 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.622604 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.622605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.622605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.622612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.624067 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.624481 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.624482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.624483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.624543 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.624795 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.624796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.624796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.624917 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.624989 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.625264 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.625358 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:24:59.625627 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.616775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:42409 (local address 127.0.0.1:54724)
0808 02:24:59.616923 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.616963 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.617400 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.617623 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.617626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.617871 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.618134 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.618141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.618817 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.618818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.619762 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.619765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.620170 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.620171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.620334 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.620790 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.620800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.622488 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.624164 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.624172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.624174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.624397 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.624619 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.624620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.624620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.624723 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.624995 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.625003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.625134 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.625231 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.625382 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260808 02:24:59.626219 12944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.60@KRBTEST.COM" created.
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:24:59.633755 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.626508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44236)
0808 02:24:59.626590 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.626602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.626647 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.626846 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.626860 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.626967 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.627012 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.627014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.627838 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.627839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.628470 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.628472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.629077 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.629078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.629131 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.629557 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.629562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.630682 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.632771 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.632772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.632773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.632935 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.633159 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.633159 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.633160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.633208 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.633536 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.633537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.633569 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.633619 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.633683 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 02:24:59.633950 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.626545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.30.160.61:45205)
0808 02:24:59.626703 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.626708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.626729 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:24:59.626754 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.626756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.626766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.626835 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.627076 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.627749 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.628553 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.629001 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.629080 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.630764 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.630772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.630772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.630780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.632692 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.633017 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.633021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.633021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.633089 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.633269 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.633270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.633270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.633468 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.633562 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.633658 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.633788 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
I20260808 02:24:59.634656 12944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.635015 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.634872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46024)
0808 02:24:59.634929 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.634986 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 02:24:59.635176 12888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
I20260808 02:24:59.635627 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.635532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46040)
0808 02:24:59.635567 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.635600 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260808 02:24:59.635753 12944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:38353: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab.
Entry for principal kudu/127.30.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab.
I20260808 02:24:59.646198 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:38353
--webserver_interface=127.30.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab
--principal=kudu/127.30.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:42409,127.30.160.61:45205,127.30.160.60:38353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:24:59.682876 12862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } attempt: 3
I20260808 02:24:59.684993 12944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } attempt: 1
I20260808 02:24:59.685359 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.685232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46056)
0808 02:24:59.685283 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.685320 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:24:59.685745 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.685616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46066)
0808 02:24:59.685678 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.685716 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 02:24:59.685936 12944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:38353: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:24:59.690294 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.683410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.30.160.61:45205)
0808 02:24:59.683472 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.683474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.683514 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:24:59.683539 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.683540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.683548 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.683591 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.684060 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.684563 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.685584 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.685877 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.685945 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.687769 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.687770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.687771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.687779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.689175 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.689527 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.689528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.689529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.689585 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.689782 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.689783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.689784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.689901 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.689973 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.690134 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.690208 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:24:59.690300 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.683330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44244)
0808 02:24:59.683403 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.683415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.683468 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.683635 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.683637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.683854 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.683968 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.683971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.684646 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.684647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.685479 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.685483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.685956 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.685957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.686044 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.686515 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.686519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.687681 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.689259 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.689260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.689261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.689440 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.689656 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.689656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.689657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.689718 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.689975 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.689977 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.690022 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.690109 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.690208 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 02:24:59.690699 12862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.691022 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.690919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46082)
0808 02:24:59.690968 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.690996 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:24:59.691327 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.691243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46098)
0808 02:24:59.691284 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.691307 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 02:24:59.691416 12862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:38353: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
I20260808 02:24:59.722623 12862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } attempt: 1
I20260808 02:24:59.723023 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.722906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46104)
0808 02:24:59.722964 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.722999 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 02:24:59.723380 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.723243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46108)
0808 02:24:59.723307 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.723349 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 02:24:59.723526 12862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:38353: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
W20260808 02:24:59.730790 12951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:59.730965 12951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:59.731012 12951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:59.732198 12951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:59.732239 12951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:59.732254 12951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:59.732298 12951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:59.732322 12951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:59.733865 12951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:42409,127.30.160.61:45205,127.30.160.60:38353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab
--principal=kudu/127.30.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:38353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:24:59.734107 12951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:59.734333 12951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:59.739010 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.739051 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.739022 12956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:59.739244 12951 server_base.cc:1034] running on GCE node
I20260808 02:24:59.739472 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:59.739796 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:59.740964 12951 hybrid_clock.cc:648] HybridClock initialized: now 1786155899740945 us; error 31 us; skew 500 ppm
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:59.743559 12951 init.cc:377] Logged in from keytab as kudu/127.30.160.60@KRBTEST.COM (short username kudu)
I20260808 02:24:59.744129 12951 webserver.cc:460] Webserver started at http://127.30.160.60:37539/ using document root <none> and password file <none>
I20260808 02:24:59.744351 12951 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:59.744419 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:59.744585 12951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:59.745602 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data/instance:
uuid: "5e41c255489f4f549340eedcbd3bd833"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "50e7a300975c400fb6f99bf544b3d9a9"
server_key_iv: "7ad85cab2e0b52453a5bb9ff4b8f0812"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.745983 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal/instance:
uuid: "5e41c255489f4f549340eedcbd3bd833"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "50e7a300975c400fb6f99bf544b3d9a9"
server_key_iv: "7ad85cab2e0b52453a5bb9ff4b8f0812"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.747330 12951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:24:59.748078 12966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.748248 12951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:24:59.748358 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal
uuid: "5e41c255489f4f549340eedcbd3bd833"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-zkpd"
server_key: "50e7a300975c400fb6f99bf544b3d9a9"
server_key_iv: "7ad85cab2e0b52453a5bb9ff4b8f0812"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.748430 12951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:59.754076 12944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } attempt: 2
I20260808 02:24:59.754467 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.754344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46118)
0808 02:24:59.754402 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.754437 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:24:59.754805 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.754729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46128)
0808 02:24:59.754765 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.754789 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 02:24:59.754913 12944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:38353: Network error: Client connection negotiation failed: client connection to 127.30.160.60:38353: connect: Connection refused (error 111)
I20260808 02:24:59.762017 12951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:59.764314 12951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:59.764424 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:59.768344 12951 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.60:38353
I20260808 02:24:59.768375 13020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.60:38353 every 8 connection(s)
I20260808 02:24:59.768705 12951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
I20260808 02:24:59.769569 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:59.771011 13021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.771481 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 12951
I20260808 02:24:59.771590 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155746594248-31360-0/minicluster-data/master-2/wal/instance
I20260808 02:24:59.771842 31360 external_mini_cluster.cc:1428] Setting key 7acd892abd766a259cd3b1df6e99f383
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:24:59.780568 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.771737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.30.160.62:42409)
0808 02:24:59.771794 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.771797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.772078 (+ 281us) server_negotiation.cc:407] Connection header received
0808 02:24:59.772705 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.772707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.772720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.772771 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.773286 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.773789 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.774675 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.774885 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.774958 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.777193 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.777195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.777195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.777201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.779061 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.779470 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.779472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.779473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.779534 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.779824 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.779825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.779825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.779998 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.780082 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.780387 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.780468 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:24:59.780678 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.771718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:42409 (local address 127.0.0.1:54736)
0808 02:24:59.771927 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.771976 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.772621 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.772787 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.772790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.773044 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.773191 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.773195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.773886 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.773888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.774584 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.774586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.774978 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.774979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.775113 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.775599 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.775609 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.777113 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.779139 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.779145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.779147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.779396 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.779606 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.779606 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.779606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.779746 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.780087 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.780093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.780243 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.780366 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.780484 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260808 02:24:59.780982 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.772308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.30.160.60:38353)
0808 02:24:59.772457 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.772462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.772526 (+ 64us) server_negotiation.cc:407] Connection header received
0808 02:24:59.772645 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.772650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.772668 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.772773 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.773381 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.774180 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.774913 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.775207 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.775297 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.777785 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.777793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.777794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.777801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.779627 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.780059 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.780062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.780063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.780121 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.780340 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.780340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.780341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.780549 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.780716 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.780804 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.780878 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260808 02:24:59.781431 13021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:24:59.788667 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.781685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:45205 (local address 127.0.0.1:44248)
0808 02:24:59.781739 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.781751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.781789 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.781972 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.781973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.782126 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.782180 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.782181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.782935 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.782936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.783494 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.783496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.783978 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.783979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.784050 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.784532 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.784536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.785821 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.787591 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.787592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.787593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.787768 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.788010 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.788011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.788011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.788096 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.788349 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.788350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.788395 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.788449 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.788551 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260808 02:24:59.788682 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.781797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.30.160.61:45205)
0808 02:24:59.781836 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.781838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.781846 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:24:59.781886 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.781887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.781897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.781953 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.782261 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.782863 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.783569 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.783868 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.783977 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.785911 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.785913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.785913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.785921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.787490 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.787849 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.787851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.787851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.787936 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.788166 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.788167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.788167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.788266 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.788348 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.788497 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.788570 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:24:59.789038 13021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:24:59.789404 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.781954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.30.160.62:42409)
0808 02:24:59.781991 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.781992 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.782000 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:24:59.782019 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.782020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.782029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.782077 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.782586 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.783075 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.783769 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.784019 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.784088 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.786506 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.786508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.786508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.786515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.788013 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.788354 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.788355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.788355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.788406 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.788628 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.788629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.788629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.788748 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.788934 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.789139 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.789207 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:24:59.795370 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.789263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46148)
0808 02:24:59.789300 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.789310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.789342 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.789474 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.789475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.789622 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.789654 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.789655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.790374 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.790375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.790822 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.790823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.791192 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.791193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.791241 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.791622 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.791627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.792802 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.794385 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.794386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.794387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.794563 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.794754 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.794755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.794756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.794808 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.795056 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.795057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.795092 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.795158 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.795248 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 02:24:59.795382 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.789295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.30.160.60:38353)
0808 02:24:59.789337 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.789339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.789348 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:24:59.789390 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.789391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.789401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.789460 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.789714 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.790298 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.790913 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.791114 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.791192 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.792913 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.792914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.792915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.792921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.794305 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.794632 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.794633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.794634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.794688 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.794890 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.794890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.794891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.794990 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.795063 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.795181 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.795298 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":22}
I20260808 02:24:59.797026 13021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: Bootstrap starting.
I20260808 02:24:59.797506 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.790257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.30.160.61:45205)
0808 02:24:59.790298 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.790300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.790310 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:24:59.790373 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.790374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.790384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.790432 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.790902 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.791321 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.791988 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.792184 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.792264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.794429 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.794430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.794431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.794436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.796259 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.796641 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.796642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.796642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.796690 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.796935 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.796936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.796936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.797047 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.797268 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.797300 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.797394 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:24:59.798059 13021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:59.798413 13021 log.cc:824] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:59.798660 12862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } attempt: 2
I20260808 02:24:59.799297 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:24:59.799409 13021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: No bootstrap required, opened a new log
I20260808 02:24:59.801702 13021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:24:59.801898 13021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:59.801966 13021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e41c255489f4f549340eedcbd3bd833, State: Initialized, Role: FOLLOWER
I20260808 02:24:59.802145 13021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:24:59.802517 13021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:24:59.804569 13036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:24:59.804726 13036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:24:59.805558 13048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:24:59.805660 13048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:24:59.806814 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.798994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.30.160.60:38353)
0808 02:24:59.799098 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.799099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.799107 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:24:59.799127 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.799127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.799136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.799490 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.799521 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.800187 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.801076 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.801363 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.801432 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.803588 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.803590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.803590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.803597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.805557 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.805951 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.805953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.805953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.806016 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.806246 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.806247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.806248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.806405 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.806472 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.806662 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.806731 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:24:59.806821 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.798959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46154)
0808 02:24:59.799003 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.799012 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.799046 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.799225 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.799227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.799375 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.799414 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.799416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.800301 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.800303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.800996 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.800998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.801542 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.801543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.801605 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.802070 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.802075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.803495 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.805648 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.805649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.805650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.805872 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.806104 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.806105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.806105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.806171 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.806501 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.806502 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.806546 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.806617 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.806713 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 02:24:59.804658 12951 master_runner.cc:186] Error getting master registration for 127.30.160.62:42409: OK, instance_id { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" instance_seqno: 1786155899453531 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:24:59.807710 12951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:24:59.809201 12862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: Bootstrap starting.
I20260808 02:24:59.809337 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.801492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.30.160.62:42409)
0808 02:24:59.801552 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.801554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.801562 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:24:59.801583 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.801584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.801592 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.801649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.802427 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.803095 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.804481 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.804740 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.805095 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.806327 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.806328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.806328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.806334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.807979 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.808376 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.808377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.808377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.808431 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.808616 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.808616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.808617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.808722 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.809112 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.809150 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.809215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:24:59.809979 12862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:59.810303 12862 log.cc:824] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:59.811201 12862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: No bootstrap required, opened a new log
I20260808 02:24:59.813663 12862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:24:59.813879 12862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:59.813998 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.801361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.30.160.61:45205)
0808 02:24:59.801404 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.801407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.801772 (+ 365us) server_negotiation.cc:407] Connection header received
0808 02:24:59.801863 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.801864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.801874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.801905 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.802847 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.803528 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.809315 (+ 5787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.809555 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.809588 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.810927 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.810929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.810929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.810936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.812517 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.812914 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.812915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.812916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.812973 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.813169 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.813170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.813170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.813291 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.813401 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.813826 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.813911 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:24:59.813984 12862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1809e9ab0d7c44ceba887238003e3c03, State: Initialized, Role: FOLLOWER
I20260808 02:24:59.814136 12862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:24:59.814465 12862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:59.814751 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.800822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.30.160.60:38353)
0808 02:24:59.801529 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.801531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.801538 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:24:59.801554 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.801555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.801572 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.801633 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.806906 (+ 5273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.807905 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.808903 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.809183 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.810992 (+ 1809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.811037 (+ 45us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.811038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.811038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.811043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.813246 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.813643 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.813644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.813645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.813726 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.813964 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.813965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.813965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.814086 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.814515 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.814558 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.814641 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":76,"threads_started":1}
I20260808 02:24:59.815950 13052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:24:59.816083 13052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:24:59.816277 13063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:24:59.816339 13063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:24:59.816887 12791 master_runner.cc:186] Error getting master registration for 127.30.160.61:45205: OK, instance_id { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" instance_seqno: 1786155899612135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:24:59.819782 12791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:24:59.846089 12944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } attempt: 3
Aug 08 02:24:59 dist-test-slave-zkpd krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:24:59.854820 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.846383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:38353 (local address 127.0.0.1:46170)
0808 02:24:59.846439 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.846451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.846493 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.846648 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.846649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.846865 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.846923 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.846925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.848181 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.848182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.849000 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.849002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.849522 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.849523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.849585 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.850111 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.850116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.851667 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.853665 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.853666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.853666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.853906 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.854140 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.854141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.854141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.854207 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.854502 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.854503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.854549 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.854621 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.854716 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:24:59.854857 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.846472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.30.160.60:38353)
0808 02:24:59.846519 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.846521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.846530 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:24:59.846552 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.846553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.846563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.846626 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.847067 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.848089 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.849102 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.849439 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.849514 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.851804 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.851806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.851806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.851814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.853573 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.853986 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.853988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.853988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.854064 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.854277 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.854278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.854279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.854429 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.854501 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.854665 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.854757 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:24:59.857054 12944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: Bootstrap starting.
I20260808 02:24:59.857895 12944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:59.858214 12944 log.cc:824] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:59.859524 12944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: No bootstrap required, opened a new log
I20260808 02:24:59.861537 12944 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:24:59.861677 12944 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:59.861712 12944 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9c6cee03654a0f8b4a437a3c01fd5f, State: Initialized, Role: FOLLOWER
I20260808 02:24:59.861866 12944 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:24:59.862181 12944 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:59.862138 13066 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:24:59.862332 13066 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:24:59.864048 13078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:24:59.864148 13078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:24:59.866103 12870 master_runner.cc:263] No leader master found from master aa9c6cee03654a0f8b4a437a3c01fd5f
I20260808 02:24:59.866149 12870 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 02:24:59.951310 12812 tablet.cc:2369] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:25:00.014981 12972 tablet.cc:2369] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:00.109824 13036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 02:25:00.109886 12892 tablet.cc:2369] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:00.109977 13036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:25:00.110262 13036 leader_election.cc:288] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1809e9ab0d7c44ceba887238003e3c03 (127.30.160.62:42409), aa9c6cee03654a0f8b4a437a3c01fd5f (127.30.160.61:45205)
I20260808 02:25:00.110512 12834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e41c255489f4f549340eedcbd3bd833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1809e9ab0d7c44ceba887238003e3c03" is_pre_election: true
I20260808 02:25:00.110641 12834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e41c255489f4f549340eedcbd3bd833 in term 0.
I20260808 02:25:00.110595 12914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e41c255489f4f549340eedcbd3bd833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" is_pre_election: true
I20260808 02:25:00.110795 12914 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e41c255489f4f549340eedcbd3bd833 in term 0.
I20260808 02:25:00.110852 12970 leader_election.cc:302] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1809e9ab0d7c44ceba887238003e3c03, 5e41c255489f4f549340eedcbd3bd833; no voters:
I20260808 02:25:00.111018 13036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:00.111078 13036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:00.111100 13036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:00.111866 13036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:25:00.112026 13036 leader_election.cc:288] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [CANDIDATE]: Term 1 election: Requested vote from peers 1809e9ab0d7c44ceba887238003e3c03 (127.30.160.62:42409), aa9c6cee03654a0f8b4a437a3c01fd5f (127.30.160.61:45205)
I20260808 02:25:00.112166 12914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e41c255489f4f549340eedcbd3bd833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f"
I20260808 02:25:00.112177 12834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e41c255489f4f549340eedcbd3bd833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1809e9ab0d7c44ceba887238003e3c03"
I20260808 02:25:00.112257 12834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:00.112260 12914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:00.112970 12914 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e41c255489f4f549340eedcbd3bd833 in term 1.
I20260808 02:25:00.113054 12834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e41c255489f4f549340eedcbd3bd833 in term 1.
I20260808 02:25:00.113144 12967 leader_election.cc:302] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e41c255489f4f549340eedcbd3bd833, aa9c6cee03654a0f8b4a437a3c01fd5f; no voters:
I20260808 02:25:00.113255 13036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:00.113369 13036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [term 1 LEADER]: Becoming Leader. State: Replica: 5e41c255489f4f549340eedcbd3bd833, State: Running, Role: LEADER
I20260808 02:25:00.113482 13036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } }
I20260808 02:25:00.113940 13080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e41c255489f4f549340eedcbd3bd833. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.114032 13080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:00.114250 13082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:00.114456 13082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:00.115929 12834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [term 1 FOLLOWER]: Refusing update from remote peer 5e41c255489f4f549340eedcbd3bd833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:00.115929 12914 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [term 1 FOLLOWER]: Refusing update from remote peer 5e41c255489f4f549340eedcbd3bd833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:00.116276 13080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:00.116408 13036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:00.117609 13066 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e41c255489f4f549340eedcbd3bd833. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.117698 13066 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:00.117682 13052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e41c255489f4f549340eedcbd3bd833. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.117774 13052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:00.118497 13036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.118531 13080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.118584 13036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:00.118628 13080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:00.118631 13084 mvcc.cc:205] Tried to move back new op lower bound from 7316094566872264704 to 7316094566865362944. Current Snapshot: MvccSnapshot[applied={T|T < 7316094566872264704}]
I20260808 02:25:00.118686 13052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.118763 13052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:00.118929 13082 catalog_manager.cc:1347] Generated new cluster ID: 49f75e5611a44eb3b7ea97a499f6c8e1
I20260808 02:25:00.118978 13082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:00.118985 13086 mvcc.cc:205] Tried to move back new op lower bound from 7316094566872264704 to 7316094566865362944. Current Snapshot: MvccSnapshot[applied={T|T < 7316094566872264704}]
I20260808 02:25:00.119274 13066 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e41c255489f4f549340eedcbd3bd833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1809e9ab0d7c44ceba887238003e3c03" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 42409 } } peers { permanent_uuid: "aa9c6cee03654a0f8b4a437a3c01fd5f" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 45205 } } peers { permanent_uuid: "5e41c255489f4f549340eedcbd3bd833" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 38353 } } }
I20260808 02:25:00.119338 13066 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:00.124166 13082 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:00.124632 13082 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:00.127820 13082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e41c255489f4f549340eedcbd3bd833: Generated new TSK 0
I20260808 02:25:00.127978 13082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:25:00.816787 13063 catalog_manager.cc:1259] Loaded cluster ID: 49f75e5611a44eb3b7ea97a499f6c8e1
I20260808 02:25:00.816871 13063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: loading cluster ID for follower catalog manager: success
I20260808 02:25:00.817624 13063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: acquiring CA information for follower catalog manager: success
I20260808 02:25:00.817906 13063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1809e9ab0d7c44ceba887238003e3c03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:25:00.864632 13078 catalog_manager.cc:1259] Loaded cluster ID: 49f75e5611a44eb3b7ea97a499f6c8e1
I20260808 02:25:00.864737 13078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: loading cluster ID for follower catalog manager: success
I20260808 02:25:00.865581 13078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: acquiring CA information for follower catalog manager: success
I20260808 02:25:00.865927 13078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa9c6cee03654a0f8b4a437a3c01fd5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:25:01.196092 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12791
I20260808 02:25:01.203172 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12870
I20260808 02:25:01.209146 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 12951
2026-08-08T02:25:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3149 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:01 dist-test-slave-zkpd krb5kdc[13098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:01 dist-test-slave-zkpd krb5kdc[13098](info): set up 2 sockets
Aug 08 02:25:01 dist-test-slave-zkpd krb5kdc[13098](info): commencing operation
krb5kdc: starting...
W20260808 02:25:02.270030 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:25:02 dist-test-slave-zkpd krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:25:02.393980 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:33977
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:02.475054 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:02.475266 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:02.475289 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:02.476464 13111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:02.476506 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:02.476521 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:02.476562 13111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:02.476593 13111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:02.478020 13111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:33977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:33977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:02.478266 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:02.478525 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:02.483227 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:02.483234 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:02.483287 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:02.483538 13111 server_base.cc:1034] running on GCE node
I20260808 02:25:02.483808 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:02.484120 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:02.485294 13111 hybrid_clock.cc:648] HybridClock initialized: now 1786155902485264 us; error 44 us; skew 500 ppm
Aug 08 02:25:02 dist-test-slave-zkpd krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:02.487838 13111 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:25:02.488344 13111 webserver.cc:460] Webserver started at http://127.30.160.62:38507/ using document root <none> and password file <none>
I20260808 02:25:02.488523 13111 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:02.488562 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:02.488649 13111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:02.489625 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d"
format_stamp: "Formatted at 2026-08-08 02:25:02 on dist-test-slave-zkpd"
server_key: "d82433359f7b6512db1449c793b15881"
server_key_iv: "8a3829648983afe7b2b2b3ec09f36009"
server_key_version: "encryptionkey@0"
I20260808 02:25:02.489920 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d"
format_stamp: "Formatted at 2026-08-08 02:25:02 on dist-test-slave-zkpd"
server_key: "d82433359f7b6512db1449c793b15881"
server_key_iv: "8a3829648983afe7b2b2b3ec09f36009"
server_key_version: "encryptionkey@0"
I20260808 02:25:02.491130 13111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:25:02.491840 13126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:02.492038 13111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:02.492146 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d"
format_stamp: "Formatted at 2026-08-08 02:25:02 on dist-test-slave-zkpd"
server_key: "d82433359f7b6512db1449c793b15881"
server_key_iv: "8a3829648983afe7b2b2b3ec09f36009"
server_key_version: "encryptionkey@0"
I20260808 02:25:02.492219 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:02.503116 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:02.505455 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:02.505616 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:02.509634 13111 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:33977
I20260808 02:25:02.509668 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:33977 every 8 connection(s)
I20260808 02:25:02.510001 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:02.510838 13183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:02.513109 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d: Bootstrap starting.
I20260808 02:25:02.513676 13183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:02.513979 13183 log.cc:824] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:02.514644 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d: No bootstrap required, opened a new log
I20260808 02:25:02.516227 13183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33977 } }
I20260808 02:25:02.516376 13183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:02.516420 13183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14528f67e7fa4ed3b0bfd7e1dd0c4e2d, State: Initialized, Role: FOLLOWER
I20260808 02:25:02.516568 13183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33977 } }
I20260808 02:25:02.516628 13183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:02.516649 13183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:02.516713 13183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:02.517217 13183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33977 } }
I20260808 02:25:02.517333 13183 leader_election.cc:302] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14528f67e7fa4ed3b0bfd7e1dd0c4e2d; no voters:
I20260808 02:25:02.517520 13183 leader_election.cc:288] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:02.517568 13186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:02.517760 13186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [term 1 LEADER]: Becoming Leader. State: Replica: 14528f67e7fa4ed3b0bfd7e1dd0c4e2d, State: Running, Role: LEADER
I20260808 02:25:02.517764 13183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:02.517947 13186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33977 } }
I20260808 02:25:02.518509 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33977 } } }
I20260808 02:25:02.518613 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:02.518710 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13111
I20260808 02:25:02.518810 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:25:02.518973 31360 external_mini_cluster.cc:1428] Setting key f20e191fb5514f38f13e63edb99b72ab
I20260808 02:25:02.519508 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14528f67e7fa4ed3b0bfd7e1dd0c4e2d. Latest consensus state: current_term: 1 leader_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14528f67e7fa4ed3b0bfd7e1dd0c4e2d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33977 } } }
I20260808 02:25:02.519696 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:02.519950 13198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:02.520670 13198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:02.522228 13198 catalog_manager.cc:1347] Generated new cluster ID: 300c9751ed294a10910a1ed00d433413
I20260808 02:25:02.522277 13198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:25:02 dist-test-slave-zkpd krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:02.531061 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.519599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.30.160.62:33977)
0808 02:25:02.520060 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.520070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.520099 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:25:02.520228 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.520235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.520296 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.520405 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.521234 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.522781 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.523828 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.524489 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.524602 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.527443 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.527447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.527447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.527458 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.529378 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.529801 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.529804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.529805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.529916 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.530139 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.530139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.530140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.530426 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.530547 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.530708 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.530839 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":85,"threads_started":1}
I20260808 02:25:02.532163 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:25:02.532876 13198 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:02.533444 13198 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:02.537024 13198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14528f67e7fa4ed3b0bfd7e1dd0c4e2d: Generated new TSK 0
I20260808 02:25:02.537177 13198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:25:02.586342 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13111
2026-08-08T02:25:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1377 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:02 dist-test-slave-zkpd krb5kdc[13214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:02 dist-test-slave-zkpd krb5kdc[13214](info): set up 2 sockets
Aug 08 02:25:02 dist-test-slave-zkpd krb5kdc[13214](info): commencing operation
krb5kdc: starting...
W20260808 02:25:03.643723 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:25:03 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:25:03.763298 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44323
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:44323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:03.843786 13229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:03.844048 13229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:03.844086 13229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:03.845207 13229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:03.845286 13229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:03.845309 13229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:03.845329 13229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:03.845357 13229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:03.846772 13229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:44323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:03.847035 13229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:03.847260 13229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:25:03.852222 13229 server_base.cc:1034] running on GCE node
W20260808 02:25:03.852200 13234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:03.852380 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:03.852195 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:03.852655 13229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:03.852938 13229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:03.854076 13229 hybrid_clock.cc:648] HybridClock initialized: now 1786155903854066 us; error 28 us; skew 500 ppm
Aug 08 02:25:03 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:03.856523 13229 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:25:03.857007 13229 webserver.cc:460] Webserver started at http://127.30.160.62:37607/ using document root <none> and password file <none>
I20260808 02:25:03.857182 13229 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:03.857220 13229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:03.857307 13229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:03.858322 13229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "d9ef43d0d87845b798d55c2d0cc52107"
format_stamp: "Formatted at 2026-08-08 02:25:03 on dist-test-slave-zkpd"
server_key: "3537159e79e0d521c8b70dbe7ff4b1cd"
server_key_iv: "1b575dd98a46dfe0aae70329623ad3ef"
server_key_version: "encryptionkey@0"
I20260808 02:25:03.858618 13229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "d9ef43d0d87845b798d55c2d0cc52107"
format_stamp: "Formatted at 2026-08-08 02:25:03 on dist-test-slave-zkpd"
server_key: "3537159e79e0d521c8b70dbe7ff4b1cd"
server_key_iv: "1b575dd98a46dfe0aae70329623ad3ef"
server_key_version: "encryptionkey@0"
I20260808 02:25:03.859841 13229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:25:03.860581 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:03.860733 13229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:25:03.860824 13229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "d9ef43d0d87845b798d55c2d0cc52107"
format_stamp: "Formatted at 2026-08-08 02:25:03 on dist-test-slave-zkpd"
server_key: "3537159e79e0d521c8b70dbe7ff4b1cd"
server_key_iv: "1b575dd98a46dfe0aae70329623ad3ef"
server_key_version: "encryptionkey@0"
I20260808 02:25:03.860882 13229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:03.872939 13229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:03.874640 13229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:03.874743 13229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:03.878618 13229 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:44323
I20260808 02:25:03.878690 13307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:44323 every 8 connection(s)
I20260808 02:25:03.879014 13229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:03.880049 13308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:03.882203 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap starting.
I20260808 02:25:03.882759 13308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:03.883051 13308 log.cc:824] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:03.883751 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: No bootstrap required, opened a new log
I20260808 02:25:03.885352 13308 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:03.885506 13308 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:03.885557 13308 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ef43d0d87845b798d55c2d0cc52107, State: Initialized, Role: FOLLOWER
I20260808 02:25:03.885694 13308 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:03.885775 13308 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:03.885818 13308 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:03.885875 13308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:03.886648 13308 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:03.886777 13308 leader_election.cc:302] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ef43d0d87845b798d55c2d0cc52107; no voters:
I20260808 02:25:03.886969 13308 leader_election.cc:288] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:03.887006 13311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:03.887164 13311 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 LEADER]: Becoming Leader. State: Replica: d9ef43d0d87845b798d55c2d0cc52107, State: Running, Role: LEADER
I20260808 02:25:03.887225 13308 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:03.887308 13311 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:03.887703 13312 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ef43d0d87845b798d55c2d0cc52107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } } }
I20260808 02:25:03.887799 13312 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:03.887712 13314 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ef43d0d87845b798d55c2d0cc52107. Latest consensus state: current_term: 1 leader_uuid: "d9ef43d0d87845b798d55c2d0cc52107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } } }
I20260808 02:25:03.887878 13314 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:03.888116 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13229
I20260808 02:25:03.888240 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:25:03.888319 13321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:03.888446 31360 external_mini_cluster.cc:1428] Setting key 1f1d3fb453caff0be29d279455de9be7
I20260808 02:25:03.888847 13321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:03.890614 13321 catalog_manager.cc:1347] Generated new cluster ID: db1edc7beb1846c3835bc600a1b3441c
I20260808 02:25:03.890669 13321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:25:03 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:03.900619 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.888953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.30.160.62:44323)
0808 02:25:03.889702 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.889710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.889732 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:25:03.889856 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.889863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.889937 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.890090 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.890889 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.892300 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.893390 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.893981 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.894070 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.896930 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.896934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.896934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.896950 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.898948 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.899326 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.899329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.899329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.899441 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.899639 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.899640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.899640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.899986 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.900134 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.900243 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.900391 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":708,"thread_start_us":86,"threads_started":1}
I20260808 02:25:03.905936 13321 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:03.906412 13321 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:03.913028 13321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Generated new TSK 0
I20260808 02:25:03.913190 13321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:25:03.939420 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44323
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:04.021951 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:04.022147 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:04.022171 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:04.023310 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:04.023351 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:04.023376 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:25:04.024946 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:04.025202 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:04.025460 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:04.030318 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.030341 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.030400 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.030541 13336 server_base.cc:1034] running on GCE node
I20260808 02:25:04.030746 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:04.030942 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:04.032075 13336 hybrid_clock.cc:648] HybridClock initialized: now 1786155904032064 us; error 30 us; skew 500 ppm
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:04.034535 13336 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:25:04.034904 13336 webserver.cc:460] Webserver started at http://127.30.160.1:38059/ using document root <none> and password file <none>
I20260808 02:25:04.035089 13336 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:04.035130 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:04.035215 13336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:04.036257 13336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "9746f2cf816d4751a6df5cbacbf3995a"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "117eb71934568e6ffd9d1fe799f7295b"
server_key_iv: "d46e6c3dc47c2903c47bebaa9651a77c"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.036561 13336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "9746f2cf816d4751a6df5cbacbf3995a"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "117eb71934568e6ffd9d1fe799f7295b"
server_key_iv: "d46e6c3dc47c2903c47bebaa9651a77c"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.037740 13336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:25:04.038384 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.038559 13336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:25:04.038662 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "9746f2cf816d4751a6df5cbacbf3995a"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "117eb71934568e6ffd9d1fe799f7295b"
server_key_iv: "d46e6c3dc47c2903c47bebaa9651a77c"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.038734 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:04.061822 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:04.063866 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:04.064072 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:04.064319 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:04.064653 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:04.064706 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.064746 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:04.064776 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.070462 13336 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:34083
I20260808 02:25:04.070554 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:34083 every 8 connection(s)
I20260808 02:25:04.070845 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:04.074493 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13336
I20260808 02:25:04.074617 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:04.074836 31360 external_mini_cluster.cc:1428] Setting key 3b549d331e7ca445d7b735cdb3dd0371
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:04.080883 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.071627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:58395 (local address 127.30.160.62:44323)
0808 02:25:04.071694 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.071696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.071910 (+ 214us) server_negotiation.cc:407] Connection header received
0808 02:25:04.072429 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.072430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.072443 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.072497 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:04.073418 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.073931 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.075339 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.075554 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.075641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.077942 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:04.077944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:04.077944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:04.077952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:04.079321 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.079793 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.079794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.079795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.079861 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.080188 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.080189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.080189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.080306 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:04.080385 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.080666 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.080776 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:25:04.081012 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.071594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:58395)
0808 02:25:04.071768 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.071809 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.072345 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.072564 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.072567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.073150 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:04.073345 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.073350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.074015 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.074016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.075217 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.075220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.075655 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.075656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.075789 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.076342 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:04.076351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:04.077841 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:04.079414 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:04.079421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:04.079424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:04.079712 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:04.079999 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:04.080000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:04.080001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:04.080116 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:04.080380 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:04.080387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:04.080527 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.080631 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.080789 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260808 02:25:04.081581 13472 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44323
I20260808 02:25:04.081692 13472 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:04.081897 13472 heartbeater.cc:511] Master 127.30.160.62:44323 requested a full tablet report, sending...
I20260808 02:25:04.082481 13265 ts_manager.cc:194] Registered new tserver with Master: 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1:34083)
I20260808 02:25:04.083079 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:58395
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:25:04.102489 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44323
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:04.185146 13477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:04.185339 13477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:04.185364 13477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:04.186415 13477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:04.186450 13477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:04.186475 13477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:25:04.187990 13477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:04.188241 13477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:04.188426 13477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:04.193178 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.193235 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.193241 13477 server_base.cc:1034] running on GCE node
W20260808 02:25:04.193189 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.193612 13477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:04.193912 13477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:04.195077 13477 hybrid_clock.cc:648] HybridClock initialized: now 1786155904195063 us; error 27 us; skew 500 ppm
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:04.197465 13477 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:25:04.197891 13477 webserver.cc:460] Webserver started at http://127.30.160.2:37491/ using document root <none> and password file <none>
I20260808 02:25:04.198124 13477 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:04.198190 13477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:04.198340 13477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:04.199333 13477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "8fb9f5f6eb4afa45d35a1e5beb5fcd35"
server_key_iv: "31690c675b23fb4599b290b2ee44698f"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.199697 13477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "8fb9f5f6eb4afa45d35a1e5beb5fcd35"
server_key_iv: "31690c675b23fb4599b290b2ee44698f"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.201066 13477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:04.201763 13492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.201946 13477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:25:04.202032 13477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "8fb9f5f6eb4afa45d35a1e5beb5fcd35"
server_key_iv: "31690c675b23fb4599b290b2ee44698f"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.202116 13477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:04.208374 13477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:04.210321 13477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:04.210500 13477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:04.210743 13477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:04.211099 13477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:04.211158 13477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.211210 13477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:04.211239 13477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.218190 13477 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:43013
I20260808 02:25:04.218223 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:43013 every 8 connection(s)
I20260808 02:25:04.218568 13477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:04.227243 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.219321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:33837 (local address 127.30.160.62:44323)
0808 02:25:04.219372 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.219374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.219553 (+ 179us) server_negotiation.cc:407] Connection header received
0808 02:25:04.220098 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.220100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.220110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.220173 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:04.221072 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.221621 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.222564 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.222769 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.222831 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.224896 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:04.224897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:04.224898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:04.224904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:04.226080 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.226428 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.226429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.226429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.226478 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.226680 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.226680 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.226681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.226817 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:04.226902 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.227110 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.227177 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:25:04.227389 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.219255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:33837)
0808 02:25:04.219425 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.219465 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.220002 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.220246 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.220250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.220792 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:04.220987 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.220991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.221705 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.221706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.222475 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.222477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.222854 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.222855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.222951 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.223453 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:04.223461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:04.224799 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:04.226151 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:04.226157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:04.226159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:04.226366 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:04.226537 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:04.226537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:04.226538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:04.226629 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:04.226888 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:04.226892 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:04.226997 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.227077 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.227187 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260808 02:25:04.227515 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13477
I20260808 02:25:04.227663 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:04.227883 31360 external_mini_cluster.cc:1428] Setting key a593dfdcc160d06ff9703471c175e71f
I20260808 02:25:04.227929 13610 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44323
I20260808 02:25:04.228044 13610 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:04.228258 13610 heartbeater.cc:511] Master 127.30.160.62:44323 requested a full tablet report, sending...
I20260808 02:25:04.228677 13265 ts_manager.cc:194] Registered new tserver with Master: 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013)
I20260808 02:25:04.229154 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:33837
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:25:04.254518 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44323
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:04.336182 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:04.336414 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:04.336452 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:04.337579 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:04.337649 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:04.337694 13615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:25:04.339129 13615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:04.339403 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:04.339634 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:04.344672 13620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.344663 13623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.344664 13621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.344884 13615 server_base.cc:1034] running on GCE node
I20260808 02:25:04.345109 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:04.345347 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:04.346518 13615 hybrid_clock.cc:648] HybridClock initialized: now 1786155904346512 us; error 35 us; skew 500 ppm
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:04.349043 13615 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:25:04.349486 13615 webserver.cc:460] Webserver started at http://127.30.160.3:43459/ using document root <none> and password file <none>
I20260808 02:25:04.349733 13615 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:04.349802 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:04.349956 13615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:04.350987 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "17659e07a9824b658f5128afd67063bb"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "a358b8e126847bd20dcbda96f5b919fa"
server_key_iv: "d667cab24fe4cc6eeb968fac3bf9406f"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.351343 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "17659e07a9824b658f5128afd67063bb"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "a358b8e126847bd20dcbda96f5b919fa"
server_key_iv: "d667cab24fe4cc6eeb968fac3bf9406f"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.352721 13615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:25:04.353427 13630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.353588 13615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:25:04.353693 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "17659e07a9824b658f5128afd67063bb"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "a358b8e126847bd20dcbda96f5b919fa"
server_key_iv: "d667cab24fe4cc6eeb968fac3bf9406f"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.353782 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:04.376207 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:04.378059 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:04.378227 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:04.378463 13615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:04.378778 13615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:04.378830 13615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.378875 13615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:04.378904 13615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.384917 13615 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:43577
I20260808 02:25:04.385010 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:43577 every 8 connection(s)
I20260808 02:25:04.385311 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:04.389936 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13615
I20260808 02:25:04.390074 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:04.390251 31360 external_mini_cluster.cc:1428] Setting key 897292cb0cae51f827e1f0bcdf9333d0
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:04.394579 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.386020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:38143 (local address 127.30.160.62:44323)
0808 02:25:04.386062 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.386064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.386261 (+ 197us) server_negotiation.cc:407] Connection header received
0808 02:25:04.386757 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.386758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.386767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.386816 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:04.387839 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.388549 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.389549 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.389769 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.389845 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.392097 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:04.392099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:04.392099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:04.392107 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:04.393215 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.393617 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.393619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.393619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.393682 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.393955 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.393956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.393956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.394066 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:04.394141 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.394410 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.394487 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:25:04.394775 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.385981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:38143)
0808 02:25:04.386142 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.386184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.386673 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.386877 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.386880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.387525 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:04.387731 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.387737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.388643 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.388644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.389442 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.389444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.389867 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.389868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.389990 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.390518 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:04.390526 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:04.392008 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:04.393298 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:04.393313 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:04.393315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:04.393537 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:04.393764 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:04.393765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:04.393766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:04.393880 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:04.394133 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:04.394140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:04.394275 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.394388 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.394540 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260808 02:25:04.395259 13750 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44323
I20260808 02:25:04.395355 13750 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:04.395485 13750 heartbeater.cc:511] Master 127.30.160.62:44323 requested a full tablet report, sending...
I20260808 02:25:04.395828 13265 ts_manager.cc:194] Registered new tserver with Master: 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577)
I20260808 02:25:04.396198 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:38143
I20260808 02:25:04.401822 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:25:04.408553 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.403089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.30.160.62:44323)
0808 02:25:04.403180 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.403182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.403229 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:25:04.403279 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.403280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.403287 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.403334 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:04.403843 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.404346 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.405088 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.405261 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.405314 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.406354 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:04.406355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:04.406355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:04.406361 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:04.407547 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.407849 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.407850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.407850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.407943 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:04.408120 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:04.408121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:04.408121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:04.408249 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:04.408327 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.408413 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.408468 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:25:04.409917 13265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:04.411427 13265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:04.417794 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.413757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.30.160.3:43577)
0808 02:25:04.414319 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.414323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.414332 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:25:04.414367 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.414369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.414385 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.414462 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:04.414573 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.415581 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.416600 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.417330 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.417440 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.417512 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.417653 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.417689 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":117,"threads_started":1}
I20260808 02:25:04.417794 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.413538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43577 (local address 127.0.0.1:36522)
0808 02:25:04.413847 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.413858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.413913 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.414477 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.414478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.414480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:04.414513 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.414514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.415685 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.415686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.416493 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.416496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.417442 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.417443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.417517 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.417530 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.417623 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.417705 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260808 02:25:04.417794 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.413535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:34083 (local address 127.0.0.1:43512)
0808 02:25:04.413680 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.413735 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.413797 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.414173 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.414176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.414179 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:04.414241 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.414243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.415195 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.415196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.416529 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.416532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.417422 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.417423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.417517 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.417531 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.417623 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.417705 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":35,"thread_start_us":59,"threads_started":1}
I20260808 02:25:04.418056 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.413547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43013 (local address 127.0.0.1:42070)
0808 02:25:04.413917 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.413927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.413970 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.414190 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.414191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.414194 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:04.414253 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.414255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.415542 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.415543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.416648 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.416650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.417641 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.417642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.417702 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.417706 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.417761 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.417813 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":33,"thread_start_us":44,"threads_started":1}
I20260808 02:25:04.418056 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.413642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.30.160.1:34083)
0808 02:25:04.413998 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.414004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.414014 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:25:04.414052 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.414054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.414068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.414169 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:04.414308 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.415093 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.416623 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.417258 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.417324 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.417389 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.417927 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.417976 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":104,"threads_started":1}
I20260808 02:25:04.418495 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 0434f6498dd944abbda35b5624d0a27d (DEFAULT_TABLE table=test-table [id=1d4d458b640140a2b4d6c1252cd14827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:04.418766 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0434f6498dd944abbda35b5624d0a27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:04.418671 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 0434f6498dd944abbda35b5624d0a27d (DEFAULT_TABLE table=test-table [id=1d4d458b640140a2b4d6c1252cd14827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:04.418987 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0434f6498dd944abbda35b5624d0a27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:04.421442 13764 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Bootstrap starting.
I20260808 02:25:04.421553 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.413778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.30.160.2:43013)
0808 02:25:04.414033 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.414037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.414046 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:25:04.414077 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.414079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.414090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.414177 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:04.414339 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.415445 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.416728 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.417437 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.417509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.417571 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.421423 (+ 3852us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.421472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260808 02:25:04.421921 13765 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Bootstrap starting.
I20260808 02:25:04.422209 13764 tablet_bootstrap.cc:654] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:04.422168 13541 tablet_service.cc:1467] Processing CreateTablet for tablet 0434f6498dd944abbda35b5624d0a27d (DEFAULT_TABLE table=test-table [id=1d4d458b640140a2b4d6c1252cd14827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:04.422499 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0434f6498dd944abbda35b5624d0a27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:04.422574 13765 tablet_bootstrap.cc:654] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:04.422935 13764 log.cc:824] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:04.423206 13765 log.cc:824] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:04.423772 13764 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: No bootstrap required, opened a new log
I20260808 02:25:04.423846 13764 ts_tablet_manager.cc:1397] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:04.424031 13765 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: No bootstrap required, opened a new log
I20260808 02:25:04.424098 13765 ts_tablet_manager.cc:1397] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:04.425434 13768 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Bootstrap starting.
I20260808 02:25:04.425896 13764 raft_consensus.cc:348] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.426076 13764 raft_consensus.cc:374] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:04.426014 13765 raft_consensus.cc:348] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.426120 13764 raft_consensus.cc:729] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17659e07a9824b658f5128afd67063bb, State: Initialized, Role: FOLLOWER
I20260808 02:25:04.426149 13765 raft_consensus.cc:374] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:04.426195 13765 raft_consensus.cc:729] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9746f2cf816d4751a6df5cbacbf3995a, State: Initialized, Role: FOLLOWER
I20260808 02:25:04.426255 13764 consensus_queue.cc:260] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.426301 13765 consensus_queue.cc:260] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.426338 13768 tablet_bootstrap.cc:654] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:04.426548 13764 ts_tablet_manager.cc:1428] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:04.426548 13765 ts_tablet_manager.cc:1428] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:25:04.426597 13472 heartbeater.cc:503] Master 127.30.160.62:44323 was elected leader, sending a full tablet report...
I20260808 02:25:04.426680 13750 heartbeater.cc:503] Master 127.30.160.62:44323 was elected leader, sending a full tablet report...
I20260808 02:25:04.426920 13768 log.cc:824] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:04.427839 13768 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: No bootstrap required, opened a new log
I20260808 02:25:04.427942 13768 ts_tablet_manager.cc:1397] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:25:04.429271 13768 raft_consensus.cc:348] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.429421 13768 raft_consensus.cc:374] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:04.429468 13768 raft_consensus.cc:729] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d9db86011f4d8c85ffe1698c6e3aae, State: Initialized, Role: FOLLOWER
I20260808 02:25:04.429579 13768 consensus_queue.cc:260] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.429875 13768 ts_tablet_manager.cc:1428] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:04.429895 13610 heartbeater.cc:503] Master 127.30.160.62:44323 was elected leader, sending a full tablet report...
W20260808 02:25:04.469146 13611 tablet.cc:2369] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:25:04.571637 13473 tablet.cc:2369] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:25:04.635984 13751 tablet.cc:2369] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:04.675937 13770 raft_consensus.cc:482] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:04.676056 13770 raft_consensus.cc:504] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.676373 13770 leader_election.cc:288] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577), 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013)
I20260808 02:25:04.680583 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.676735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:34635 (local address 127.30.160.2:43013)
0808 02:25:04.676776 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.676778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.676789 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:25:04.676878 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.676879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.676889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.676945 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:04.677163 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.677927 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.679569 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.680198 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.680267 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.680275 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.680469 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.680507 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:25:04.680583 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.676699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:35941 (local address 127.30.160.3:43577)
0808 02:25:04.676748 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.676751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.676792 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:25:04.676878 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.676879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.676889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.676945 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:04.677163 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.677922 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.679569 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.680183 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.680266 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.680275 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.680469 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.680507 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:25:04.680620 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.676587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43577 (local address 127.30.160.1:35941)
0808 02:25:04.676732 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.676741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.676794 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.676967 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.676968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.676971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:04.677078 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.677080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.678018 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.678018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.679459 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.679463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.680275 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.680277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.680380 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.680387 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.680445 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.680509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":106,"thread_start_us":63,"threads_started":1}
I20260808 02:25:04.680624 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.676587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43013 (local address 127.30.160.1:34635)
0808 02:25:04.676725 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.676736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.676794 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.676967 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.676968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:04.676971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:04.677083 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.677085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.678018 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.678019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.679459 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:04.679463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.680277 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.680278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:04.680380 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.680385 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:25:04.680445 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:25:04.680509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":142,"thread_start_us":65,"threads_started":1}
I20260808 02:25:04.681057 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0434f6498dd944abbda35b5624d0a27d" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" is_pre_election: true
I20260808 02:25:04.681167 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0434f6498dd944abbda35b5624d0a27d" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17659e07a9824b658f5128afd67063bb" is_pre_election: true
I20260808 02:25:04.681227 13562 raft_consensus.cc:2454] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 0.
I20260808 02:25:04.681298 13702 raft_consensus.cc:2454] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 0.
I20260808 02:25:04.681470 13352 leader_election.cc:302] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d9db86011f4d8c85ffe1698c6e3aae, 9746f2cf816d4751a6df5cbacbf3995a; no voters:
I20260808 02:25:04.681597 13770 raft_consensus.cc:2781] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:04.681658 13770 raft_consensus.cc:482] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:04.681696 13770 raft_consensus.cc:3037] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:04.682417 13770 raft_consensus.cc:504] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.682559 13770 leader_election.cc:288] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 election: Requested vote from peers 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577), 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013)
I20260808 02:25:04.682706 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0434f6498dd944abbda35b5624d0a27d" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
I20260808 02:25:04.682706 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0434f6498dd944abbda35b5624d0a27d" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17659e07a9824b658f5128afd67063bb"
I20260808 02:25:04.682785 13562 raft_consensus.cc:3037] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:04.682785 13702 raft_consensus.cc:3037] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:04.683682 13702 raft_consensus.cc:2454] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 1.
I20260808 02:25:04.683682 13562 raft_consensus.cc:2454] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 1.
I20260808 02:25:04.683856 13352 leader_election.cc:302] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d9db86011f4d8c85ffe1698c6e3aae, 9746f2cf816d4751a6df5cbacbf3995a; no voters:
I20260808 02:25:04.683977 13770 raft_consensus.cc:2781] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:04.684147 13770 raft_consensus.cc:686] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 LEADER]: Becoming Leader. State: Replica: 9746f2cf816d4751a6df5cbacbf3995a, State: Running, Role: LEADER
I20260808 02:25:04.684262 13770 consensus_queue.cc:237] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:04.685082 13265 catalog_manager.cc:5495] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1). New cstate: current_term: 1 leader_uuid: "9746f2cf816d4751a6df5cbacbf3995a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:04.699529 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.697107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.30.160.1:34083)
0808 02:25:04.697145 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.697147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.697156 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:25:04.697248 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.697249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.697264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.697360 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:04.697506 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.697966 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.698744 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:04.698938 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:04.699016 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:04.699191 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:04.699284 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:04.699379 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:04.699421 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 02:25:04.699441 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 02:25:04.701357 13702 raft_consensus.cc:1261] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Refusing update from remote peer 9746f2cf816d4751a6df5cbacbf3995a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:04.701651 13562 raft_consensus.cc:1261] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Refusing update from remote peer 9746f2cf816d4751a6df5cbacbf3995a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:04.701726 13776 consensus_queue.cc:1035] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:04.701939 13776 consensus_queue.cc:1035] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:04.704283 13779 mvcc.cc:205] Tried to move back new op lower bound from 7316094585654272000 to 7316094585587421184. Current Snapshot: MvccSnapshot[applied={T|T < 7316094585654272000}]
I20260808 02:25:04.704578 13781 mvcc.cc:205] Tried to move back new op lower bound from 7316094585654272000 to 7316094585587421184. Current Snapshot: MvccSnapshot[applied={T|T < 7316094585654272000}]
I20260808 02:25:04.705099 13780 mvcc.cc:205] Tried to move back new op lower bound from 7316094585654272000 to 7316094585587421184. Current Snapshot: MvccSnapshot[applied={T|T < 7316094585654272000}]
I20260808 02:25:04.706840 13264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:04.706918 13264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:04.707690 13264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Sending DeleteTablet for 3 replicas of tablet 0434f6498dd944abbda35b5624d0a27d
I20260808 02:25:04.708057 13541 tablet_service.cc:1514] Processing DeleteTablet for tablet 0434f6498dd944abbda35b5624d0a27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:04 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42070
I20260808 02:25:04.708061 13680 tablet_service.cc:1514] Processing DeleteTablet for tablet 0434f6498dd944abbda35b5624d0a27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:04 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:36522
I20260808 02:25:04.708118 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 0434f6498dd944abbda35b5624d0a27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:04 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:43512
I20260808 02:25:04.708289 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13336
I20260808 02:25:04.708351 13791 tablet_replica.cc:330] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: stopping tablet replica
I20260808 02:25:04.710764 13790 tablet_replica.cc:330] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: stopping tablet replica
I20260808 02:25:04.710892 13790 raft_consensus.cc:2229] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:04.711030 13790 raft_consensus.cc:2258] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:04.711225 13788 tablet_replica.cc:330] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: stopping tablet replica
I20260808 02:25:04.711354 13788 raft_consensus.cc:2229] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:04.711474 13788 raft_consensus.cc:2258] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:04.711668 13790 ts_tablet_manager.cc:1905] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:04.712991 13788 ts_tablet_manager.cc:1905] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:04.713567 13790 ts_tablet_manager.cc:1918] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:04.713658 13790 log.cc:1196] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/0434f6498dd944abbda35b5624d0a27d
I20260808 02:25:04.713929 13790 ts_tablet_manager.cc:1939] T 0434f6498dd944abbda35b5624d0a27d P 35d9db86011f4d8c85ffe1698c6e3aae: Deleting consensus metadata
I20260808 02:25:04.714331 13247 catalog_manager.cc:4841] TS 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013): tablet 0434f6498dd944abbda35b5624d0a27d (table test-table [id=1d4d458b640140a2b4d6c1252cd14827]) successfully deleted
I20260808 02:25:04.714926 13788 ts_tablet_manager.cc:1918] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:04.715006 13788 log.cc:1196] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/0434f6498dd944abbda35b5624d0a27d
I20260808 02:25:04.715286 13788 ts_tablet_manager.cc:1939] T 0434f6498dd944abbda35b5624d0a27d P 17659e07a9824b658f5128afd67063bb: Deleting consensus metadata
I20260808 02:25:04.715687 13248 catalog_manager.cc:4841] TS 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577): tablet 0434f6498dd944abbda35b5624d0a27d (table test-table [id=1d4d458b640140a2b4d6c1252cd14827]) successfully deleted
W20260808 02:25:04.717689 13251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:34083 (error 108)
I20260808 02:25:04.717869 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13477
I20260808 02:25:04.718911 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.718781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:34083 (local address 127.0.0.1:43526)
0808 02:25:04.718824 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.718874 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:34083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 02:25:04.719012 13251 catalog_manager.cc:4584] TS 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1:34083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0434f6498dd944abbda35b5624d0a27d: Network error: Client connection negotiation failed: client connection to 127.30.160.1:34083: connect: Connection refused (error 111)
I20260808 02:25:04.728094 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13615
I20260808 02:25:04.734007 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13229
I20260808 02:25:04.740281 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44323
--webserver_interface=127.30.160.62
--webserver_port=37607
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:44323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:04.824357 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:04.824559 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:04.824582 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:04.825762 13793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:04.825804 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:04.825819 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:04.825831 13793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:04.825843 13793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:04.827329 13793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:44323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:04.827553 13793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:04.827857 13793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:04.837692 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.837663 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.837765 13793 server_base.cc:1034] running on GCE node
W20260808 02:25:04.837669 13799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.838163 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:04.838514 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:04.839682 13793 hybrid_clock.cc:648] HybridClock initialized: now 1786155904839676 us; error 35 us; skew 500 ppm
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:04.842249 13793 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:25:04.842682 13793 webserver.cc:460] Webserver started at http://127.30.160.62:37607/ using document root <none> and password file <none>
I20260808 02:25:04.842937 13793 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:04.843024 13793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:04.844571 13793 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 02:25:04.845240 13809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.845482 13793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:04.845592 13793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "d9ef43d0d87845b798d55c2d0cc52107"
format_stamp: "Formatted at 2026-08-08 02:25:03 on dist-test-slave-zkpd"
server_key: "3537159e79e0d521c8b70dbe7ff4b1cd"
server_key_iv: "1b575dd98a46dfe0aae70329623ad3ef"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.845999 13793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:04.856917 13793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:04.858649 13793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:04.858841 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:04.862860 13793 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:44323
I20260808 02:25:04.862911 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:44323 every 8 connection(s)
I20260808 02:25:04.863271 13793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:04.864916 13864 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:25:04.865562 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap starting.
I20260808 02:25:04.865816 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13793
I20260808 02:25:04.866112 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:34083
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:44323
--webserver_port=38059
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:04.875164 13864 log.cc:824] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:04.877914 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:04.878270 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap complete.
I20260808 02:25:04.882323 13864 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:04.882625 13864 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ef43d0d87845b798d55c2d0cc52107, State: Initialized, Role: FOLLOWER
I20260808 02:25:04.882776 13864 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:04.882836 13864 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:04.882896 13864 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:04.882943 13864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:25:04.884245 13864 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:04.884351 13864 leader_election.cc:302] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ef43d0d87845b798d55c2d0cc52107; no voters:
I20260808 02:25:04.884500 13864 leader_election.cc:288] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:25:04.884563 13868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:25:04.884713 13868 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 LEADER]: Becoming Leader. State: Replica: d9ef43d0d87845b798d55c2d0cc52107, State: Running, Role: LEADER
I20260808 02:25:04.884732 13864 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:04.884810 13868 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:04.885159 13869 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9ef43d0d87845b798d55c2d0cc52107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } } }
I20260808 02:25:04.885236 13869 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:04.885463 13871 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ef43d0d87845b798d55c2d0cc52107. Latest consensus state: current_term: 2 leader_uuid: "d9ef43d0d87845b798d55c2d0cc52107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } } }
I20260808 02:25:04.885591 13871 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:04.885637 13877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:04.886418 13877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:04.886531 13877 catalog_manager.cc:1259] Loaded cluster ID: db1edc7beb1846c3835bc600a1b3441c
I20260808 02:25:04.886797 13877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:04.887059 13877 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 02:25:04.891724 13877 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 02:25:04.894232 13877 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:04.894452 13877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Loaded TSK: 0
I20260808 02:25:04.894675 13877 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:25:04.957125 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:04.957358 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:04.957401 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:04.958560 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:04.958624 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:04.958667 13866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:25:04.960197 13866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:34083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=38059
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:04.960457 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:04.960690 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:04.965633 13893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.965777 13894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:04.965637 13896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:04.965938 13866 server_base.cc:1034] running on GCE node
I20260808 02:25:04.966135 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:04.966399 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:04.967557 13866 hybrid_clock.cc:648] HybridClock initialized: now 1786155904967533 us; error 38 us; skew 500 ppm
Aug 08 02:25:04 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:04.970095 13866 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:25:04.970613 13866 webserver.cc:460] Webserver started at http://127.30.160.1:38059/ using document root <none> and password file <none>
I20260808 02:25:04.970856 13866 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:04.970916 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:04.972385 13866 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:25:04.973117 13904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:04.973261 13866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:04.973380 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "9746f2cf816d4751a6df5cbacbf3995a"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "117eb71934568e6ffd9d1fe799f7295b"
server_key_iv: "d46e6c3dc47c2903c47bebaa9651a77c"
server_key_version: "encryptionkey@0"
I20260808 02:25:04.973726 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:04.983549 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:04.986063 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:04.986248 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:04.986522 13866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:04.987036 13911 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:25:04.988524 13866 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:25:04.988574 13866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 02:25:04.988610 13866 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:25:04.989272 13866 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:25:04.989318 13866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 02:25:04.989362 13911 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Bootstrap starting.
I20260808 02:25:04.997115 13866 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:34083
I20260808 02:25:04.997287 14027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:34083 every 8 connection(s)
I20260808 02:25:04.997620 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:05.001215 13911 log.cc:824] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:05.003670 13911 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:05.004021 13911 tablet_bootstrap.cc:492] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Bootstrap complete.
I20260808 02:25:05.005199 13911 ts_tablet_manager.cc:1397] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260808 02:25:05.005815 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 13866
I20260808 02:25:05.006126 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:43013
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:44323
--webserver_port=37491
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:05.007488 13911 raft_consensus.cc:348] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:05.007948 13911 raft_consensus.cc:729] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9746f2cf816d4751a6df5cbacbf3995a, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.008121 13911 consensus_queue.cc:260] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } } peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } }
I20260808 02:25:05.008404 13911 ts_tablet_manager.cc:1428] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 02:25:05.011003 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.998338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:40383)
0808 02:25:04.998611 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.998649 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:04.999197 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:04.999765 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:04.999769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.000135 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:05.000385 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.000391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.002089 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.002090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.003345 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.003348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.004187 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.004188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.004342 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.004967 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:05.004978 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:05.006829 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:05.009471 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.009475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.009477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.009686 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.010001 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.010002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.010002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.010096 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.010557 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:05.010562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:05.010666 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.010743 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.010863 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260808 02:25:05.011135 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.998462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:40383 (local address 127.30.160.62:44323)
0808 02:25:04.998672 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:04.998683 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:04.998728 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:25:04.999463 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:04.999471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:04.999543 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:04.999724 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.000522 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.001965 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.003452 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.004085 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.004194 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.006946 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.006950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.006950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.006962 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.009387 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.009776 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.009779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.009779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.009915 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.010163 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.010164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.010164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.010485 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.010572 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.010824 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.010955 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260808 02:25:05.011793 14028 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44323
I20260808 02:25:05.011935 14028 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:05.012108 14028 heartbeater.cc:511] Master 127.30.160.62:44323 requested a full tablet report, sending...
I20260808 02:25:05.012884 13827 ts_manager.cc:194] Registered new tserver with Master: 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1:34083)
I20260808 02:25:05.014068 13827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:40383
I20260808 02:25:05.014936 14028 heartbeater.cc:503] Master 127.30.160.62:44323 was elected leader, sending a full tablet report...
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:25:05.023322 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.013472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:34083 (local address 127.0.0.1:43532)
0808 02:25:05.013634 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.013667 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.013735 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.014611 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.014616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.014912 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:05.014961 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.014963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.016280 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.016281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.017198 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.017201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.017831 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.017832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.017904 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.018404 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:05.018409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:05.020104 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:05.022232 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.022238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.022240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.022433 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.022661 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.022661 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.022662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.022715 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.023052 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:05.023058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:05.023098 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.023159 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.023245 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260808 02:25:05.023435 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.014331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.30.160.1:34083)
0808 02:25:05.014465 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.014469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.014478 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:25:05.014504 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.014506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.014537 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.014607 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.015106 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.016186 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.017298 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.017703 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.017771 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.020200 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.020209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.020209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.020217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.022147 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.022516 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.022520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.022520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.022588 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.022778 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.022779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.022779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.022980 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.023073 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.023203 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.023310 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260808 02:25:05.023810 13954 tablet_service.cc:1514] Processing DeleteTablet for tablet 0434f6498dd944abbda35b5624d0a27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:04 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:43532
I20260808 02:25:05.024541 14037 tablet_replica.cc:330] stopping tablet replica
I20260808 02:25:05.024653 14037 raft_consensus.cc:2229] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:05.024745 14037 raft_consensus.cc:2258] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.025076 14037 ts_tablet_manager.cc:1905] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.026997 14037 ts_tablet_manager.cc:1918] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.027061 14037 log.cc:1196] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/0434f6498dd944abbda35b5624d0a27d
I20260808 02:25:05.027289 14037 ts_tablet_manager.cc:1939] T 0434f6498dd944abbda35b5624d0a27d P 9746f2cf816d4751a6df5cbacbf3995a: Deleting consensus metadata
I20260808 02:25:05.027786 13813 catalog_manager.cc:4841] TS 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1:34083): tablet 0434f6498dd944abbda35b5624d0a27d (table test-table [id=1d4d458b640140a2b4d6c1252cd14827]) successfully deleted
W20260808 02:25:05.100071 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:05.100312 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:05.100358 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:05.101505 14034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:05.101572 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:05.101615 14034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:25:05.103329 14034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:43013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=37491
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:05.103595 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:05.103825 14034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:05.108611 14042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.108672 14043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.108672 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.108875 14034 server_base.cc:1034] running on GCE node
I20260808 02:25:05.109084 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:05.109367 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:05.110510 14034 hybrid_clock.cc:648] HybridClock initialized: now 1786155905110497 us; error 39 us; skew 500 ppm
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:05.112946 14034 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:25:05.113404 14034 webserver.cc:460] Webserver started at http://127.30.160.2:37491/ using document root <none> and password file <none>
I20260808 02:25:05.113637 14034 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:05.113736 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:05.115027 14034 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.115576 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.115746 14034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:05.115842 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "8fb9f5f6eb4afa45d35a1e5beb5fcd35"
server_key_iv: "31690c675b23fb4599b290b2ee44698f"
server_key_version: "encryptionkey@0"
I20260808 02:25:05.116235 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:05.146422 14034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:05.148484 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:05.148648 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:05.148891 14034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:05.149250 14034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:05.149302 14034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.149344 14034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:05.149385 14034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.155278 14034 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:43013
I20260808 02:25:05.155335 14175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:43013 every 8 connection(s)
I20260808 02:25:05.155669 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:05.160851 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14034
I20260808 02:25:05.161144 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:43577
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:44323
--webserver_port=43459
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:05.165526 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.156393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:47077 (local address 127.30.160.62:44323)
0808 02:25:05.156444 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.156446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.156671 (+ 225us) server_negotiation.cc:407] Connection header received
0808 02:25:05.157221 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.157223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.157240 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.157292 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.158237 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.158741 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.159804 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.160078 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.160163 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.162640 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.162642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.162642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.162650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.164132 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.164538 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.164539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.164540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.164603 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.164856 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.164858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.164858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.164980 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.165050 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.165328 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.165435 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:25:05.165690 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.156381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:47077)
0808 02:25:05.156545 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.156581 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.157106 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.157362 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.157366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.157935 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:05.158123 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.158128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.158835 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.158836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.159712 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.159715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.160174 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.160175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.160297 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.160824 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:05.160832 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:05.162545 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:05.164226 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.164233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.164235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.164459 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.164681 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.164681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.164682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.164789 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.165053 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:05.165060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:05.165195 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.165295 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.165451 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260808 02:25:05.166172 14176 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44323
I20260808 02:25:05.166290 14176 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:05.166507 14176 heartbeater.cc:511] Master 127.30.160.62:44323 requested a full tablet report, sending...
I20260808 02:25:05.166908 13827 ts_manager.cc:194] Registered new tserver with Master: 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013)
I20260808 02:25:05.167276 13827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:47077
W20260808 02:25:05.246696 14179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:05.246892 14179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:05.246917 14179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:05.248049 14179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:05.248087 14179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:05.248114 14179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:25:05.249631 14179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:43577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=43459
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:05.249923 14179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:05.250165 14179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:05.255154 14184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.255154 14187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.255179 14185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.255385 14179 server_base.cc:1034] running on GCE node
I20260808 02:25:05.255683 14179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:05.255975 14179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:05.257146 14179 hybrid_clock.cc:648] HybridClock initialized: now 1786155905257129 us; error 31 us; skew 500 ppm
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:05.259543 14179 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:25:05.260062 14179 webserver.cc:460] Webserver started at http://127.30.160.3:43459/ using document root <none> and password file <none>
I20260808 02:25:05.260322 14179 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:05.260404 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:05.261694 14179 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.262292 14194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.262466 14179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:05.262563 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "17659e07a9824b658f5128afd67063bb"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "a358b8e126847bd20dcbda96f5b919fa"
server_key_iv: "d667cab24fe4cc6eeb968fac3bf9406f"
server_key_version: "encryptionkey@0"
I20260808 02:25:05.262910 14179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:05.277946 14179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:05.280320 14179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:05.280481 14179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:05.280721 14179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:05.281045 14179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:05.281100 14179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.281141 14179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:05.281169 14179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.287663 14179 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:43577
I20260808 02:25:05.287711 14314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:43577 every 8 connection(s)
I20260808 02:25:05.288137 14179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:05.297271 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14179
I20260808 02:25:05.297458 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.288860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:54757 (local address 127.30.160.62:44323)
0808 02:25:05.288934 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.288937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.289169 (+ 232us) server_negotiation.cc:407] Connection header received
0808 02:25:05.289655 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.289656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.289667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.289755 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.290671 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.291184 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.292249 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.292444 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.292507 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.294621 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.294623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.294623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.294632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.296020 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.296452 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.296453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.296454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.296520 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.296784 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.296784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.296785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.296912 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.296989 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.297270 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.297359 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:25:05.297673 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.288846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:54757)
0808 02:25:05.289040 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.289077 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.289562 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.289811 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.289814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.290388 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:05.290584 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.290589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.291267 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.291268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.292152 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.292154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.292523 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.292524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.292622 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.293073 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:05.293082 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:05.294510 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:05.296114 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.296122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.296125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.296370 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.296590 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:05.296590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:05.296591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:05.296707 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:05.296979 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:05.296987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:05.297131 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.297246 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.297424 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260808 02:25:05.298210 14315 heartbeater.cc:347] Connected to a master server at 127.30.160.62:44323
I20260808 02:25:05.298353 14315 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:05.298575 14315 heartbeater.cc:511] Master 127.30.160.62:44323 requested a full tablet report, sending...
I20260808 02:25:05.299003 13827 ts_manager.cc:194] Registered new tserver with Master: 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577)
I20260808 02:25:05.299389 13827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:54757
I20260808 02:25:05.304426 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.297787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.30.160.62:44323)
0808 02:25:05.297823 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.297825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.297960 (+ 135us) server_negotiation.cc:407] Connection header received
0808 02:25:05.298102 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.298102 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.298113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.298166 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.298900 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.299445 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.300455 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.300638 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.300711 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.301924 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.301925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.301925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.301932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.303211 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.303555 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.303556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.303556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.303616 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.303820 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.303820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.303821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.304000 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.304193 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.304230 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.304329 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:25:05.305258 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:25:05.308084 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.305604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58844 (local address 127.30.160.62:44323)
0808 02:25:05.305667 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.305669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.305742 (+ 73us) server_negotiation.cc:407] Connection header received
0808 02:25:05.305839 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.305839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.305879 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.305958 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:05.306070 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.306519 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.307300 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.307498 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.307581 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.307745 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:05.307849 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:05.307967 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.307996 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.308013 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:25:05.308367 13827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:05.309962 13827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:05.312197 13954 tablet_service.cc:1467] Processing CreateTablet for tablet 3c92067a4492454aaf8fd98be1617282 (DEFAULT_TABLE table=test-table [id=d3d82da3394f442daa183d5b8fbd0ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:05.312423 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c92067a4492454aaf8fd98be1617282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:05.314123 13911 tablet_bootstrap.cc:492] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Bootstrap starting.
I20260808 02:25:05.314468 13911 tablet_bootstrap.cc:654] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:05.315390 13911 tablet_bootstrap.cc:492] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: No bootstrap required, opened a new log
I20260808 02:25:05.315460 13911 ts_tablet_manager.cc:1397] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:05.315618 13911 raft_consensus.cc:348] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.315701 13911 raft_consensus.cc:374] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:05.315753 13911 raft_consensus.cc:729] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9746f2cf816d4751a6df5cbacbf3995a, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.315799 13911 consensus_queue.cc:260] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.315985 13911 ts_tablet_manager.cc:1428] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.315973 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.311846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43013 (local address 127.0.0.1:42084)
0808 02:25:05.311935 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.311944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.311980 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.312490 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.312491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.312493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.312535 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.312536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.313478 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.313479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.314683 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.314685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.315666 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.315667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.315745 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.315770 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.315804 (+ 34us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.315861 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 02:25:05.316111 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.311953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43577 (local address 127.0.0.1:36530)
0808 02:25:05.312074 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.312086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.312135 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.312662 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.312663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.312664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.312696 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.312697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.313887 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.313888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.314908 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.314910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.315818 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.315819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.315874 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.315878 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.315946 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.315977 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260808 02:25:05.316249 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.312171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.30.160.3:43577)
0808 02:25:05.312436 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.312440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.312451 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:25:05.312551 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.312553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.312571 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.312665 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.312750 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.313775 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.314999 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.315726 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.316024 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.316098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.316126 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.316170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260808 02:25:05.316730 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.312097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.30.160.2:43013)
0808 02:25:05.312287 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.312291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.312303 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:25:05.312355 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.312357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.312373 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.312471 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.312591 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.313366 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.314782 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.315560 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.316219 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.316292 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.316568 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.316613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260808 02:25:05.317010 14244 tablet_service.cc:1467] Processing CreateTablet for tablet 3c92067a4492454aaf8fd98be1617282 (DEFAULT_TABLE table=test-table [id=d3d82da3394f442daa183d5b8fbd0ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:05.317358 14244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c92067a4492454aaf8fd98be1617282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:05.320091 14324 tablet_bootstrap.cc:492] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Bootstrap starting.
I20260808 02:25:05.320608 14105 tablet_service.cc:1467] Processing CreateTablet for tablet 3c92067a4492454aaf8fd98be1617282 (DEFAULT_TABLE table=test-table [id=d3d82da3394f442daa183d5b8fbd0ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:05.320770 14324 tablet_bootstrap.cc:654] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:05.320928 14105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c92067a4492454aaf8fd98be1617282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:05.321300 14324 log.cc:824] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:05.322026 14324 tablet_bootstrap.cc:492] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: No bootstrap required, opened a new log
I20260808 02:25:05.322113 14324 ts_tablet_manager.cc:1397] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:05.324127 14326 tablet_bootstrap.cc:492] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Bootstrap starting.
I20260808 02:25:05.324118 14324 raft_consensus.cc:348] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.324304 14324 raft_consensus.cc:374] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:05.324343 14324 raft_consensus.cc:729] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17659e07a9824b658f5128afd67063bb, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.324476 14324 consensus_queue.cc:260] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.324800 14324 ts_tablet_manager.cc:1428] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:05.324831 14315 heartbeater.cc:503] Master 127.30.160.62:44323 was elected leader, sending a full tablet report...
I20260808 02:25:05.324898 14326 tablet_bootstrap.cc:654] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:05.325526 14326 log.cc:824] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:05.326572 14326 tablet_bootstrap.cc:492] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: No bootstrap required, opened a new log
I20260808 02:25:05.326635 14326 ts_tablet_manager.cc:1397] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:25:05.328065 14326 raft_consensus.cc:348] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.328202 14326 raft_consensus.cc:374] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:05.328224 14326 raft_consensus.cc:729] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d9db86011f4d8c85ffe1698c6e3aae, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.328342 14326 consensus_queue.cc:260] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.328588 14326 ts_tablet_manager.cc:1428] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:25:05.328665 14176 heartbeater.cc:503] Master 127.30.160.62:44323 was elected leader, sending a full tablet report...
I20260808 02:25:05.403456 14033 raft_consensus.cc:482] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:05.403606 14033 raft_consensus.cc:504] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.404049 14033 leader_election.cc:288] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013), 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577)
W20260808 02:25:05.406243 14177 tablet.cc:2369] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:05.407188 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.404315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:36905 (local address 127.30.160.3:43577)
0808 02:25:05.404367 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.404369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.404540 (+ 171us) server_negotiation.cc:407] Connection header received
0808 02:25:05.404582 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.404583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.404593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.404652 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.404890 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.405409 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.406414 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.406817 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.406856 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.406862 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.407079 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.407119 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:25:05.407207 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.404262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43577 (local address 127.30.160.1:36905)
0808 02:25:05.404450 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.404461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.404534 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.404669 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.404670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.404671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.404782 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.404783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.405500 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.405501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.406286 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.406288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.406927 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.406927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.406974 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.406992 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.407057 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.407098 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":2,"thread_start_us":129,"threads_started":1}
I20260808 02:25:05.407267 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.404389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:43041 (local address 127.30.160.2:43013)
0808 02:25:05.404431 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.404433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.404443 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:25:05.404476 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.404478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.404487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.404549 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.404743 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.405271 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.406502 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.406894 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.406985 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.406994 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.407146 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.407187 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:25:05.407294 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.404248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43013 (local address 127.30.160.1:43041)
0808 02:25:05.404324 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.404336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.404395 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.404572 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.404573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.404575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.404662 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.404664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.405383 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.405384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.406394 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.406397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.406984 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.406985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.407067 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.407069 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.407139 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.407179 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":34}
I20260808 02:25:05.407649 14265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c92067a4492454aaf8fd98be1617282" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17659e07a9824b658f5128afd67063bb" is_pre_election: true
I20260808 02:25:05.407694 14125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c92067a4492454aaf8fd98be1617282" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" is_pre_election: true
I20260808 02:25:05.407796 14265 raft_consensus.cc:2454] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 0.
I20260808 02:25:05.407866 14125 raft_consensus.cc:2454] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 0.
I20260808 02:25:05.408079 13906 leader_election.cc:302] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17659e07a9824b658f5128afd67063bb, 9746f2cf816d4751a6df5cbacbf3995a; no voters:
I20260808 02:25:05.408239 14033 raft_consensus.cc:2781] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:05.408285 14033 raft_consensus.cc:482] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:05.408301 14033 raft_consensus.cc:3037] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:05.408990 14033 raft_consensus.cc:504] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.409127 14033 leader_election.cc:288] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 election: Requested vote from peers 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013), 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577)
I20260808 02:25:05.409260 14125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c92067a4492454aaf8fd98be1617282" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
I20260808 02:25:05.409348 14125 raft_consensus.cc:3037] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:05.409343 14265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c92067a4492454aaf8fd98be1617282" candidate_uuid: "9746f2cf816d4751a6df5cbacbf3995a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17659e07a9824b658f5128afd67063bb"
I20260808 02:25:05.409426 14265 raft_consensus.cc:3037] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:05.410161 14265 raft_consensus.cc:2454] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 1.
I20260808 02:25:05.410190 14125 raft_consensus.cc:2454] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9746f2cf816d4751a6df5cbacbf3995a in term 1.
I20260808 02:25:05.410384 13905 leader_election.cc:302] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d9db86011f4d8c85ffe1698c6e3aae, 9746f2cf816d4751a6df5cbacbf3995a; no voters:
I20260808 02:25:05.410498 14033 raft_consensus.cc:2781] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:05.410652 14033 raft_consensus.cc:686] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 1 LEADER]: Becoming Leader. State: Replica: 9746f2cf816d4751a6df5cbacbf3995a, State: Running, Role: LEADER
I20260808 02:25:05.410727 14033 consensus_queue.cc:237] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } }
I20260808 02:25:05.411530 13827 catalog_manager.cc:5495] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1). New cstate: current_term: 1 leader_uuid: "9746f2cf816d4751a6df5cbacbf3995a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9746f2cf816d4751a6df5cbacbf3995a" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 34083 } health_report { overall_health: HEALTHY } } }
I20260808 02:25:05.437112 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.434615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.30.160.1:34083)
0808 02:25:05.434680 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.434682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.434692 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:25:05.434769 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.434774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.434792 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.434897 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:05.435030 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.435523 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.436328 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.436534 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.436611 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.436767 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:05.436873 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:05.436958 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.436999 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.437021 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:25:05.438601 14125 raft_consensus.cc:1261] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Refusing update from remote peer 9746f2cf816d4751a6df5cbacbf3995a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:05.438602 14265 raft_consensus.cc:1261] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Refusing update from remote peer 9746f2cf816d4751a6df5cbacbf3995a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:05.438936 14033 consensus_queue.cc:1035] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d9db86011f4d8c85ffe1698c6e3aae" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:05.439108 14331 consensus_queue.cc:1035] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17659e07a9824b658f5128afd67063bb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:05.441560 14333 mvcc.cc:205] Tried to move back new op lower bound from 7316094588673855488 to 7316094588562894848. Current Snapshot: MvccSnapshot[applied={T|T < 7316094588673855488}]
I20260808 02:25:05.442327 14335 mvcc.cc:205] Tried to move back new op lower bound from 7316094588673855488 to 7316094588562894848. Current Snapshot: MvccSnapshot[applied={T|T < 7316094588673855488}]
I20260808 02:25:05.443796 13825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:05.443919 13825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:05.444684 13825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Sending DeleteTablet for 3 replicas of tablet 3c92067a4492454aaf8fd98be1617282
I20260808 02:25:05.444993 14244 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c92067a4492454aaf8fd98be1617282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:05 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:36530
I20260808 02:25:05.445034 14105 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c92067a4492454aaf8fd98be1617282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:05 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42084
I20260808 02:25:05.445223 14345 tablet_replica.cc:330] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: stopping tablet replica
I20260808 02:25:05.445247 14344 tablet_replica.cc:330] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: stopping tablet replica
I20260808 02:25:05.445317 14345 raft_consensus.cc:2229] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:05.445322 13954 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c92067a4492454aaf8fd98be1617282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:05 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:43532
I20260808 02:25:05.445340 14344 raft_consensus.cc:2229] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:05.445396 14037 tablet_replica.cc:330] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: stopping tablet replica
I20260808 02:25:05.445421 14345 raft_consensus.cc:2258] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.445432 14344 raft_consensus.cc:2258] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.445436 14037 raft_consensus.cc:2229] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:05.445509 14037 raft_consensus.cc:2258] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.445870 14344 ts_tablet_manager.cc:1905] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.445935 14037 ts_tablet_manager.cc:1905] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.445986 14345 ts_tablet_manager.cc:1905] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.447634 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13866
I20260808 02:25:05.447675 14345 ts_tablet_manager.cc:1918] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.447737 14345 log.cc:1196] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/3c92067a4492454aaf8fd98be1617282
I20260808 02:25:05.448096 14345 ts_tablet_manager.cc:1939] T 3c92067a4492454aaf8fd98be1617282 P 35d9db86011f4d8c85ffe1698c6e3aae: Deleting consensus metadata
I20260808 02:25:05.448282 14344 ts_tablet_manager.cc:1918] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.448372 14344 log.cc:1196] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/3c92067a4492454aaf8fd98be1617282
I20260808 02:25:05.448668 14344 ts_tablet_manager.cc:1939] T 3c92067a4492454aaf8fd98be1617282 P 17659e07a9824b658f5128afd67063bb: Deleting consensus metadata
I20260808 02:25:05.448838 13810 catalog_manager.cc:4841] TS 35d9db86011f4d8c85ffe1698c6e3aae (127.30.160.2:43013): tablet 3c92067a4492454aaf8fd98be1617282 (table test-table [id=d3d82da3394f442daa183d5b8fbd0ca8]) successfully deleted
I20260808 02:25:05.453336 13811 catalog_manager.cc:4841] TS 17659e07a9824b658f5128afd67063bb (127.30.160.3:43577): tablet 3c92067a4492454aaf8fd98be1617282 (table test-table [id=d3d82da3394f442daa183d5b8fbd0ca8]) successfully deleted
W20260808 02:25:05.457924 13813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:34083 (error 108)
I20260808 02:25:05.458163 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 14034
I20260808 02:25:05.460968 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.458344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:34083 (local address 127.0.0.1:43550)
0808 02:25:05.460871 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.460923 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:34083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2499}
W20260808 02:25:05.462900 13813 catalog_manager.cc:4584] TS 9746f2cf816d4751a6df5cbacbf3995a (127.30.160.1:34083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c92067a4492454aaf8fd98be1617282: Network error: Client connection negotiation failed: client connection to 127.30.160.1:34083: connect: Connection refused (error 111)
I20260808 02:25:05.468461 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 14179
I20260808 02:25:05.473858 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 13793
I20260808 02:25:05.480436 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44323
--webserver_interface=127.30.160.62
--webserver_port=37607
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:44323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:05.562091 14348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:05.562321 14348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:05.562367 14348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:05.563494 14348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:05.563563 14348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:05.563587 14348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:05.563614 14348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:05.563643 14348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:05.565089 14348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:44323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:05.565367 14348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:05.565605 14348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:05.570364 14354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.570327 14353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.570585 14348 server_base.cc:1034] running on GCE node
W20260808 02:25:05.570317 14356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.570910 14348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:05.571223 14348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:05.572382 14348 hybrid_clock.cc:648] HybridClock initialized: now 1786155905572368 us; error 43 us; skew 500 ppm
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:05.574734 14348 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:25:05.575158 14348 webserver.cc:460] Webserver started at http://127.30.160.62:37607/ using document root <none> and password file <none>
I20260808 02:25:05.575354 14348 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:05.575397 14348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:05.576715 14348 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.577338 14363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.577536 14348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:05.577620 14348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "d9ef43d0d87845b798d55c2d0cc52107"
format_stamp: "Formatted at 2026-08-08 02:25:03 on dist-test-slave-zkpd"
server_key: "3537159e79e0d521c8b70dbe7ff4b1cd"
server_key_iv: "1b575dd98a46dfe0aae70329623ad3ef"
server_key_version: "encryptionkey@0"
I20260808 02:25:05.577941 14348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:05.593724 14348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:05.595515 14348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:05.595623 14348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:05.599506 14348 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:44323
I20260808 02:25:05.599538 14417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:44323 every 8 connection(s)
I20260808 02:25:05.599925 14348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:05.601466 14418 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:25:05.602169 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap starting.
I20260808 02:25:05.605296 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14348
I20260808 02:25:05.605592 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:34083
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:44323
--webserver_port=38059
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:05.611388 14418 log.cc:824] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:05.613574 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:05.613888 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107: Bootstrap complete.
I20260808 02:25:05.616714 14418 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:05.617089 14418 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ef43d0d87845b798d55c2d0cc52107, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.617295 14418 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:05.617377 14418 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:05.617419 14418 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:05.617463 14418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:25:05.618923 14418 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:05.619035 14418 leader_election.cc:302] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ef43d0d87845b798d55c2d0cc52107; no voters:
I20260808 02:25:05.619222 14418 leader_election.cc:288] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:25:05.619257 14422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:25:05.619380 14422 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [term 3 LEADER]: Becoming Leader. State: Replica: d9ef43d0d87845b798d55c2d0cc52107, State: Running, Role: LEADER
I20260808 02:25:05.619490 14418 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:05.619472 14422 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } }
I20260808 02:25:05.619747 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ef43d0d87845b798d55c2d0cc52107. Latest consensus state: current_term: 3 leader_uuid: "d9ef43d0d87845b798d55c2d0cc52107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } } }
I20260808 02:25:05.619844 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:05.619963 14424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d9ef43d0d87845b798d55c2d0cc52107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef43d0d87845b798d55c2d0cc52107" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44323 } } }
I20260808 02:25:05.620091 14424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef43d0d87845b798d55c2d0cc52107 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:05.620110 14427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:05.620997 14427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:05.621116 14427 catalog_manager.cc:1259] Loaded cluster ID: db1edc7beb1846c3835bc600a1b3441c
I20260808 02:25:05.621351 14427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 02:25:05.621626 14427 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7f34f719ad google::LogMessage::Fail() at ??:0
@ 0x7f7f34f7488a google::LogMessage::SendToLog() at ??:0
@ 0x7f7f34f714de google::LogMessage::Flush() at ??:0
@ 0x7f7f34f72a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7f3ce960f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7f3ce9f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7f36e3f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7f36e34954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7f37c986db start_thread at ??:0
@ 0x7f7f33a6f61f clone at ??:0
W20260808 02:25:05.704558 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:05.704753 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:05.704777 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:05.705917 14420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:05.705957 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:05.705984 14420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:25:05.707437 14420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:34083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=38059
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:05.707659 14420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:05.707954 14420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:05.712781 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.712829 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.713028 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.713070 14420 server_base.cc:1034] running on GCE node
I20260808 02:25:05.713289 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:05.713524 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:05.714685 14420 hybrid_clock.cc:648] HybridClock initialized: now 1786155905714669 us; error 31 us; skew 500 ppm
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:05.717316 14420 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:25:05.717792 14420 webserver.cc:460] Webserver started at http://127.30.160.1:38059/ using document root <none> and password file <none>
I20260808 02:25:05.718015 14420 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:05.718086 14420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:05.719463 14420 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.720206 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.720397 14420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.720498 14420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "9746f2cf816d4751a6df5cbacbf3995a"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "117eb71934568e6ffd9d1fe799f7295b"
server_key_iv: "d46e6c3dc47c2903c47bebaa9651a77c"
server_key_version: "encryptionkey@0"
I20260808 02:25:05.720818 14420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:05.734164 14420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:05.736341 14420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:05.736508 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:05.736729 14420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:05.737159 14476 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260808 02:25:05.738989 14476 ts_tablet_manager.cc:1852] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 02:25:05.739400 14476 ts_tablet_manager.cc:1905] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.742002 14476 ts_tablet_manager.cc:1918] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.742100 14476 log.cc:1196] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/3c92067a4492454aaf8fd98be1617282
I20260808 02:25:05.742421 14476 ts_tablet_manager.cc:1939] T 3c92067a4492454aaf8fd98be1617282 P 9746f2cf816d4751a6df5cbacbf3995a: Deleting consensus metadata
I20260808 02:25:05.742657 14420 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 02:25:05.742717 14420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 02:25:05.742760 14420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:05.742786 14420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.749986 14420 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:34083
I20260808 02:25:05.750068 14589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:34083 every 8 connection(s)
I20260808 02:25:05.750344 14420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:05.752070 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14420
I20260808 02:25:05.752365 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:43013
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:44323
--webserver_port=37491
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:05.807737 14592 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:05.751067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:38619)
0808 02:25:05.751230 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.751273 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.751718 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.807498 (+ 55780us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260808 02:25:05.808055 14590 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:44323 (0 consecutive failures): Network error: Failed to ping master at 127.30.160.62:44323: Client connection negotiation failed: client connection to 127.30.160.62:44323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:25:05.808418 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.808293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:36375)
0808 02:25:05.808347 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.808389 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:25:05.808758 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.808671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:54439)
0808 02:25:05.808721 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.808744 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 02:25:05.808873 14590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:25:05.843165 14593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:05.843400 14593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:05.843442 14593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:05.844655 14593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:05.844717 14593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:05.844758 14593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:25:05.846201 14593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:43013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=37491
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:05.846462 14593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:05.846691 14593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:25:05.851833 14593 server_base.cc:1034] running on GCE node
W20260808 02:25:05.851780 14599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.851789 14601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.851807 14598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.852213 14593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:05.852456 14593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:05.853598 14593 hybrid_clock.cc:648] HybridClock initialized: now 1786155905853580 us; error 35 us; skew 500 ppm
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:05.856115 14593 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:25:05.856525 14593 webserver.cc:460] Webserver started at http://127.30.160.2:37491/ using document root <none> and password file <none>
I20260808 02:25:05.856745 14593 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:05.856789 14593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:05.858043 14593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.858639 14608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.858819 14593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:05.858914 14593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "35d9db86011f4d8c85ffe1698c6e3aae"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "8fb9f5f6eb4afa45d35a1e5beb5fcd35"
server_key_iv: "31690c675b23fb4599b290b2ee44698f"
server_key_version: "encryptionkey@0"
I20260808 02:25:05.859218 14593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:05.868551 14593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:05.870255 14593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:05.870409 14593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:05.870644 14593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:05.870937 14593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:05.870970 14593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.870990 14593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:05.871034 14593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.876818 14593 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:43013
I20260808 02:25:05.876919 14727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:43013 every 8 connection(s)
I20260808 02:25:05.877228 14593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:05.878252 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.877883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:32929)
0808 02:25:05.878038 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.878123 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260808 02:25:05.878580 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14593
W20260808 02:25:05.878698 14728 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:44323 (0 consecutive failures): Network error: Failed to ping master at 127.30.160.62:44323: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
I20260808 02:25:05.878873 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:43577
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:44323
--webserver_port=43459
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:05.879051 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.878919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:51661)
0808 02:25:05.878981 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.879017 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:25:05.879335 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.879262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:56699)
0808 02:25:05.879300 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.879321 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 02:25:05.879462 14728 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:25:05.960052 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:05.960249 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:05.960271 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:05.961458 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:05.961500 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:05.961525 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:25:05.963352 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:43577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=43459
--tserver_master_addrs=127.30.160.62:44323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:05.963551 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:05.963732 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:05.968626 14739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.968627 14737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:05.968672 14736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:05.968797 14731 server_base.cc:1034] running on GCE node
I20260808 02:25:05.968990 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:05.969237 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:05.970379 14731 hybrid_clock.cc:648] HybridClock initialized: now 1786155905970365 us; error 31 us; skew 500 ppm
Aug 08 02:25:05 dist-test-slave-zkpd krb5kdc[13214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:05.972813 14731 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:25:05.973238 14731 webserver.cc:460] Webserver started at http://127.30.160.3:43459/ using document root <none> and password file <none>
I20260808 02:25:05.973431 14731 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:05.973472 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:05.974697 14731 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:05.975244 14748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:05.975421 14731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:05.975528 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "17659e07a9824b658f5128afd67063bb"
format_stamp: "Formatted at 2026-08-08 02:25:04 on dist-test-slave-zkpd"
server_key: "a358b8e126847bd20dcbda96f5b919fa"
server_key_iv: "d667cab24fe4cc6eeb968fac3bf9406f"
server_key_version: "encryptionkey@0"
I20260808 02:25:05.975975 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:06.005962 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:06.007822 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:06.008008 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:06.008250 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:06.008574 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:06.008626 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:06.008662 14731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:06.008690 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:06.014644 14731 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:43577
I20260808 02:25:06.014700 14870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:43577 every 8 connection(s)
I20260808 02:25:06.015074 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:06.016176 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:06.015751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:57789)
0808 02:25:06.015938 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:06.016022 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260808 02:25:06.016583 14871 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:44323 (0 consecutive failures): Network error: Failed to ping master at 127.30.160.62:44323: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
I20260808 02:25:06.016857 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:06.016763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:41427)
0808 02:25:06.016807 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:06.016833 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 02:25:06.017089 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:06.017040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:48065)
0808 02:25:06.017060 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:06.017078 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260808 02:25:06.017194 14871 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:25:06.024777 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14731
I20260808 02:25:06.809424 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:06.809202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:59843)
0808 02:25:06.809323 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:06.809359 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260808 02:25:06.880085 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:06.879772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:41421)
0808 02:25:06.879965 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:06.880019 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260808 02:25:07.017680 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:07.017495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:44327)
0808 02:25:07.017601 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:07.017635 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260808 02:25:07.810043 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:07.809832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:40647)
0808 02:25:07.809966 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:07.809997 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260808 02:25:07.880683 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:07.880494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:52867)
0808 02:25:07.880612 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:07.880644 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260808 02:25:08.018252 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:08.018068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:52479)
0808 02:25:08.018181 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:08.018215 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260808 02:25:08.811002 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:08.810665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:41895)
0808 02:25:08.810917 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:08.810954 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":59,"threads_started":1}
I20260808 02:25:08.881276 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:08.881068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:50797)
0808 02:25:08.881207 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:08.881237 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260808 02:25:09.018863 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.018676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:53585)
0808 02:25:09.018790 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:09.018824 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260808 02:25:09.811704 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.811490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:50191)
0808 02:25:09.811617 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:09.811651 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260808 02:25:09.881930 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.881740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:51853)
0808 02:25:09.881861 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:09.881893 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260808 02:25:10.019488 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.019279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:36553)
0808 02:25:10.019414 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:10.019449 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260808 02:25:10.812371 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.812162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:59105)
0808 02:25:10.812287 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:10.812323 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260808 02:25:10.882596 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.882387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:59657)
0808 02:25:10.882516 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:10.882555 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260808 02:25:11.020097 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.019876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:47435)
0808 02:25:11.020024 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.020058 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260808 02:25:11.812992 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.812788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:48183)
0808 02:25:11.812908 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.812944 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260808 02:25:11.883234 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.883044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:46801)
0808 02:25:11.883164 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.883194 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260808 02:25:12.020700 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.020513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:36989)
0808 02:25:12.020627 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.020659 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260808 02:25:12.813643 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.813429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:55881)
0808 02:25:12.813560 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.813593 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260808 02:25:12.883960 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.883715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:44577)
0808 02:25:12.883865 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.883918 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260808 02:25:13.021286 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.021101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:60529)
0808 02:25:13.021216 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:13.021247 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260808 02:25:13.814252 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.814040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:45771)
0808 02:25:13.814173 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:13.814205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260808 02:25:13.884680 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.884452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:53319)
0808 02:25:13.884584 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:13.884639 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260808 02:25:14.021868 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:14.021679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:41813)
0808 02:25:14.021798 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:14.021832 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260808 02:25:14.814978 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:14.814732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:57415)
0808 02:25:14.814887 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:14.814928 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260808 02:25:14.885318 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:14.885104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:38069)
0808 02:25:14.885236 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:14.885266 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260808 02:25:15.022473 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:15.022262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:38571)
0808 02:25:15.022399 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:15.022434 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260808 02:25:15.815699 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:15.815450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:59317)
0808 02:25:15.815582 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:15.815642 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260808 02:25:15.885977 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:15.885697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:35231)
0808 02:25:15.885898 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:15.885934 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":57,"threads_started":1}
I20260808 02:25:16.023182 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:16.022885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:54033)
0808 02:25:16.023107 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:16.023143 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260808 02:25:16.816428 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:16.816214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:41533)
0808 02:25:16.816338 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:16.816377 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260808 02:25:16.886559 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:16.886362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:60421)
0808 02:25:16.886485 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:16.886520 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260808 02:25:17.023937 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:17.023681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:39487)
0808 02:25:17.023816 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:17.023855 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260808 02:25:17.817070 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:17.816870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:37993)
0808 02:25:17.816985 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:17.817020 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260808 02:25:17.887225 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:17.887018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:34399)
0808 02:25:17.887148 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:17.887184 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260808 02:25:18.024519 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:18.024332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:44997)
0808 02:25:18.024444 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:18.024479 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260808 02:25:18.817723 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:18.817491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:40663)
0808 02:25:18.817634 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:18.817674 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260808 02:25:18.887840 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:18.887619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:41209)
0808 02:25:18.887759 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:18.887793 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260808 02:25:19.025178 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.024954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:56897)
0808 02:25:19.025093 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:19.025132 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260808 02:25:19.818547 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.818304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:42219)
0808 02:25:19.818461 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:19.818499 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260808 02:25:19.888499 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.888287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:52881)
0808 02:25:19.888419 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:19.888455 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260808 02:25:20.025843 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:20.025610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:34885)
0808 02:25:20.025751 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:20.025785 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260808 02:25:20.819427 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:20.819167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.1:38075)
0808 02:25:20.819335 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:20.819381 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260808 02:25:20.889204 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:20.888990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.2:42045)
0808 02:25:20.889128 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:20.889160 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260808 02:25:21.026487 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:21.026284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:44323 (local address 127.30.160.3:48205)
0808 02:25:21.026416 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:21.026448 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:44323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260808 02:25:21.030707 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 14420
I20260808 02:25:21.037300 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 14593
I20260808 02:25:21.042791 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 14731
2026-08-08T02:25:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18455 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:21 dist-test-slave-zkpd krb5kdc[14924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:21 dist-test-slave-zkpd krb5kdc[14924](info): set up 2 sockets
Aug 08 02:25:21 dist-test-slave-zkpd krb5kdc[14924](info): commencing operation
krb5kdc: starting...
W20260808 02:25:22.140843 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:25:22.260452 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39831
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:22.342389 14938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:22.342573 14938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:22.342594 14938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:22.343611 14938 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:25:22.343642 14938 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:25:22.343654 14938 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:25:22.343665 14938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:22.343677 14938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:22.343690 14938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:22.343701 14938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:22.343712 14938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:22.345144 14938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:22.345321 14938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:22.345500 14938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:22.350239 14946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.350334 14938 server_base.cc:1034] running on GCE node
W20260808 02:25:22.350242 14943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.350315 14944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.350790 14938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:22.351066 14938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:22.352202 14938 hybrid_clock.cc:648] HybridClock initialized: now 1786155922352188 us; error 46 us; skew 500 ppm
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:22.354557 14938 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:25:22.354995 14938 webserver.cc:460] Webserver started at http://127.30.160.62:44709/ using document root <none> and password file <none>
I20260808 02:25:22.355312 14938 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:22.355398 14938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:22.355538 14938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:22.356645 14938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "5575f29f77b340d8bc5852418061ebb6"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "38edd528208b3693a6a8c356b95b8ce4"
server_key_iv: "e45d53538de523321bd21a1d4d5c91a5"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.357018 14938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "5575f29f77b340d8bc5852418061ebb6"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "38edd528208b3693a6a8c356b95b8ce4"
server_key_iv: "e45d53538de523321bd21a1d4d5c91a5"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.358363 14938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:25:22.359081 14953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.359319 14938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:22.359432 14938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "5575f29f77b340d8bc5852418061ebb6"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "38edd528208b3693a6a8c356b95b8ce4"
server_key_iv: "e45d53538de523321bd21a1d4d5c91a5"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.359524 14938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:22.374275 14938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:22.376454 14938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:22.376617 14938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:22.380698 14938 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39831
I20260808 02:25:22.380777 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39831 every 8 connection(s)
I20260808 02:25:22.381116 14938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:22.381985 15009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:22.384430 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap starting.
I20260808 02:25:22.384917 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 14938
I20260808 02:25:22.385008 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:25:22.385075 15009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:22.385233 31360 external_mini_cluster.cc:1428] Setting key 12c7ff020aa11cb98c82e97c9371a6ce
I20260808 02:25:22.385456 15009 log.cc:824] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:22.386214 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: No bootstrap required, opened a new log
I20260808 02:25:22.388244 15009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:22.388407 15009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:22.388459 15009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5575f29f77b340d8bc5852418061ebb6, State: Initialized, Role: FOLLOWER
I20260808 02:25:22.388610 15009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:22.388702 15009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:22.388747 15009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:22.388803 15009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:22.389323 15009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:22.389442 15009 leader_election.cc:302] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5575f29f77b340d8bc5852418061ebb6; no voters:
I20260808 02:25:22.389626 15009 leader_election.cc:288] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:22.389673 15014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:22.389839 15014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 LEADER]: Becoming Leader. State: Replica: 5575f29f77b340d8bc5852418061ebb6, State: Running, Role: LEADER
I20260808 02:25:22.389902 15009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:22.390030 15014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:22.391736 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5575f29f77b340d8bc5852418061ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } } }
I20260808 02:25:22.391782 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5575f29f77b340d8bc5852418061ebb6. Latest consensus state: current_term: 1 leader_uuid: "5575f29f77b340d8bc5852418061ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } } }
I20260808 02:25:22.392508 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:22.392520 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: This master's current role is: LEADER
W20260808 02:25:22.392073 15031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:25:22.392627 15031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:25:22.396083 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.385770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.30.160.62:39831)
0808 02:25:22.385968 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.385976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.386000 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:25:22.386440 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.386448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.386494 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.386633 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.387286 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.388302 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.389226 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.389773 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.390200 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.392874 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:22.392877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:22.392877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:22.392887 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:22.394562 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.394963 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.394966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.394966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.395050 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.395262 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.395262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.395262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.395460 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:22.395565 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.395701 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.395810 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260808 02:25:22.400017 15033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:22.400249 15033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:22.401917 15033 catalog_manager.cc:1347] Generated new cluster ID: 8ff0e8664cae4cb2aa335854c0ac1ba1
I20260808 02:25:22.401974 15033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:22.415825 15033 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:22.416559 15033 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:22.426760 15033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Generated new TSK 0
I20260808 02:25:22.426982 15033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:25:22.463155 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39831
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:22.570788 15037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:22.570976 15037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:22.570998 15037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:22.572175 15037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:22.572214 15037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:22.572240 15037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:25:22.573675 15037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:22.573935 15037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:22.574173 15037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:22.579006 15045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.579005 15043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.579209 15037 server_base.cc:1034] running on GCE node
W20260808 02:25:22.579263 15042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.579485 15037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:22.579670 15037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:22.580811 15037 hybrid_clock.cc:648] HybridClock initialized: now 1786155922580774 us; error 45 us; skew 500 ppm
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:22.583163 15037 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:25:22.583513 15037 webserver.cc:460] Webserver started at http://127.30.160.1:44985/ using document root <none> and password file <none>
I20260808 02:25:22.583676 15037 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:22.583714 15037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:22.583799 15037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:22.584770 15037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "fc03c7c7dc37b49c0fdbb353aeccbb1c"
server_key_iv: "af5631dec0a8c8f05b7d43d1aa70fe64"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.585067 15037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "fc03c7c7dc37b49c0fdbb353aeccbb1c"
server_key_iv: "af5631dec0a8c8f05b7d43d1aa70fe64"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.586298 15037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:25:22.586977 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.587153 15037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:25:22.587251 15037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "fc03c7c7dc37b49c0fdbb353aeccbb1c"
server_key_iv: "af5631dec0a8c8f05b7d43d1aa70fe64"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.587322 15037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:22.601977 15037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:22.603645 15037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:22.603796 15037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:22.604081 15037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:22.604373 15037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:22.604425 15037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.604465 15037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:22.604507 15037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.610198 15037 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42033
I20260808 02:25:22.610276 15173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42033 every 8 connection(s)
I20260808 02:25:22.610570 15037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:22.619118 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15037
I20260808 02:25:22.619269 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:22.619467 31360 external_mini_cluster.cc:1428] Setting key d629ededf61d9eb625f1997984e69136
I20260808 02:25:22.619647 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.611291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:39673 (local address 127.30.160.62:39831)
0808 02:25:22.611381 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.611384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.611578 (+ 194us) server_negotiation.cc:407] Connection header received
0808 02:25:22.612130 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.612132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.612145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.612210 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.613125 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.613630 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.614637 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.614861 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.614949 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.616983 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:22.616984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:22.616985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:22.616992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:22.618130 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.618556 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.618557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.618558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.618621 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.618895 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.618896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.618896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.619059 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:22.619138 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.619469 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.619558 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:25:22.619840 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.611224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:39673)
0808 02:25:22.611407 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.611467 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.612023 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.612277 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.612281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.612837 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.613025 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.613030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.613760 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.613761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.614539 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.614541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.614962 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.614963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.615086 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.615501 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.615508 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.616881 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:22.618216 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:22.618223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:22.618226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:22.618476 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:22.618700 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:22.618701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:22.618701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:22.618811 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:22.619129 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:22.619136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:22.619307 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 02:25:22.619430 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:25:22.619583 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260808 02:25:22.620455 15174 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39831
I20260808 02:25:22.620558 15174 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:22.620728 15174 heartbeater.cc:511] Master 127.30.160.62:39831 requested a full tablet report, sending...
I20260808 02:25:22.621299 14970 ts_manager.cc:194] Registered new tserver with Master: 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033)
I20260808 02:25:22.621920 14970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:39673
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:25:22.644959 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39831
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:22.728104 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:22.728346 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:22.728384 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:22.729504 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:22.729578 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:22.729621 15179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:25:22.731055 15179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:22.731323 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:22.731549 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:22.736406 15185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.736577 15179 server_base.cc:1034] running on GCE node
W20260808 02:25:22.736418 15186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.736415 15188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.736912 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:22.737124 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:22.738300 15179 hybrid_clock.cc:648] HybridClock initialized: now 1786155922738286 us; error 31 us; skew 500 ppm
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:22.740801 15179 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:25:22.741228 15179 webserver.cc:460] Webserver started at http://127.30.160.2:34679/ using document root <none> and password file <none>
I20260808 02:25:22.741460 15179 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:22.741526 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:22.741673 15179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:22.742624 15179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "cacf84c4da3c4ed19f2db86fd8aed978"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "eb98630dc702a32fd7d9d1e23b029a68"
server_key_iv: "30f949b26d4121b9d4bc4fc7957e73f6"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.742970 15179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "cacf84c4da3c4ed19f2db86fd8aed978"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "eb98630dc702a32fd7d9d1e23b029a68"
server_key_iv: "30f949b26d4121b9d4bc4fc7957e73f6"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.744333 15179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:25:22.745050 15196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.745220 15179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.745326 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "cacf84c4da3c4ed19f2db86fd8aed978"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "eb98630dc702a32fd7d9d1e23b029a68"
server_key_iv: "30f949b26d4121b9d4bc4fc7957e73f6"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.745401 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:22.762804 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:22.764709 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:22.764863 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:22.765082 15179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:22.765370 15179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:22.765424 15179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.765456 15179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:22.765481 15179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.771271 15179 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38365
I20260808 02:25:22.771325 15316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38365 every 8 connection(s)
I20260808 02:25:22.771644 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:22.779767 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15179
I20260808 02:25:22.779911 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:22.780104 31360 external_mini_cluster.cc:1428] Setting key c1b24927ed288905fdf3fbc81128b042
I20260808 02:25:22.780885 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.772423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:43671 (local address 127.30.160.62:39831)
0808 02:25:22.772553 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.772555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.772774 (+ 219us) server_negotiation.cc:407] Connection header received
0808 02:25:22.773289 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.773290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.773302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.773355 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.774308 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.774800 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.775868 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.776086 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.776174 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.778196 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:22.778198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:22.778198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:22.778205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:22.779392 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.779810 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.779811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.779812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.779870 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.780154 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.780155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.780156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.780305 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:22.780376 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.780710 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.780795 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:25:22.781152 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.772385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:43671)
0808 02:25:22.772628 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.772678 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.773177 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.773435 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.773439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.774026 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.774204 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.774208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.774909 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.774911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.775771 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.775774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.776183 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.776184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.776311 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.776768 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.776776 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.778093 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:22.779475 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:22.779482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:22.779484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:22.779715 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:22.779967 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:22.779968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:22.779968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:22.780085 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:22.780390 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:22.780398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:22.780541 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 02:25:22.780660 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:25:22.780888 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260808 02:25:22.781702 15317 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39831
I20260808 02:25:22.781813 15317 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:22.782001 15317 heartbeater.cc:511] Master 127.30.160.62:39831 requested a full tablet report, sending...
I20260808 02:25:22.782344 14970 ts_manager.cc:194] Registered new tserver with Master: cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365)
I20260808 02:25:22.782698 14970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:43671
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:25:22.804678 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39831
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:22.885809 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:22.886026 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:22.886062 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:22.887171 15322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:22.887208 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:22.887234 15322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:25:22.888796 15322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:22.888973 15322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:22.889135 15322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:22.894470 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.894479 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.894527 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.894706 15322 server_base.cc:1034] running on GCE node
I20260808 02:25:22.894912 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:22.895150 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:22.896314 15322 hybrid_clock.cc:648] HybridClock initialized: now 1786155922896295 us; error 33 us; skew 500 ppm
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:22.898974 15322 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:25:22.899510 15322 webserver.cc:460] Webserver started at http://127.30.160.3:44031/ using document root <none> and password file <none>
I20260808 02:25:22.899770 15322 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:22.899854 15322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:22.900028 15322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:22.901147 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "e1d6ce5e04a1485c8482a50a26d34aff"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "b6c59d4b4deceb6cde242d80e95fd653"
server_key_iv: "d4934549a44f8d81b04b88ded67b5abd"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.901558 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "e1d6ce5e04a1485c8482a50a26d34aff"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "b6c59d4b4deceb6cde242d80e95fd653"
server_key_iv: "d4934549a44f8d81b04b88ded67b5abd"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.903026 15322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:25:22.903884 15340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.904088 15322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:22.904213 15322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "e1d6ce5e04a1485c8482a50a26d34aff"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "b6c59d4b4deceb6cde242d80e95fd653"
server_key_iv: "d4934549a44f8d81b04b88ded67b5abd"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.904297 15322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:22.913149 15322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:22.914861 15322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:22.915019 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:22.915236 15322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:22.915545 15322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:22.915599 15322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.915637 15322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:22.915665 15322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.921712 15322 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:42073
I20260808 02:25:22.921779 15460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:42073 every 8 connection(s)
I20260808 02:25:22.922055 15322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:25:22 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:22.929467 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15322
I20260808 02:25:22.929560 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:22.929761 31360 external_mini_cluster.cc:1428] Setting key 9cefb76167c6c146f40e07aac375fc79
I20260808 02:25:22.931551 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.922729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:47053 (local address 127.30.160.62:39831)
0808 02:25:22.922794 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.922796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.922959 (+ 163us) server_negotiation.cc:407] Connection header received
0808 02:25:22.923417 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.923418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.923428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.923486 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.924429 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.924979 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.925912 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.926119 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.926172 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.928325 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:22.928326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:22.928327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:22.928333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:22.929937 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.930468 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.930469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.930469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.930535 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.930846 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.930847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.930848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.930992 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:22.931072 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.931361 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.931461 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:25:22.931720 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.922656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:47053)
0808 02:25:22.922840 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.922878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.923336 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.923544 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.923547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.924150 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.924337 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.924342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.925067 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.925068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.925835 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.925837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.926189 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.926190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.926287 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.926787 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.926795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.928242 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:22.930081 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:22.930087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:22.930089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:22.930366 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:22.930614 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:22.930615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:22.930616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:22.930754 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:22.931063 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:22.931070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:22.931215 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 02:25:22.931328 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:25:22.931494 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260808 02:25:22.932284 15461 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39831
I20260808 02:25:22.932370 15461 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:22.932642 15461 heartbeater.cc:511] Master 127.30.160.62:39831 requested a full tablet report, sending...
I20260808 02:25:22.933223 14970 ts_manager.cc:194] Registered new tserver with Master: e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
I20260808 02:25:22.933594 14970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:47053
I20260808 02:25:22.941769 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:25:22.949074 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.943072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.30.160.62:39831)
0808 02:25:22.943144 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.943147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.943190 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:25:22.943258 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.943259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.943268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.943319 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.943871 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.944423 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.945165 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.945356 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.945408 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.946593 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:22.946595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:22.946595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:22.946602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:22.948017 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.948327 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.948328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.948328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.948377 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.948546 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.948547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.948547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.948677 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:22.948748 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.948922 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.948979 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:25:22.950822 14970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:22.952492 14970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:22.959455 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.954946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:42073 (local address 127.0.0.1:35598)
0808 02:25:22.955154 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.955199 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.955274 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.955779 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.955783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.955786 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:22.955854 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.955857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.957237 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.957238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.958216 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.958219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.959090 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.959091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.959189 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.959204 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:25:22.959305 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:25:22.959369 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":25,"thread_start_us":59,"threads_started":1}
I20260808 02:25:22.959462 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.955068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.30.160.3:42073)
0808 02:25:22.955278 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.955284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.955295 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:25:22.955423 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.955426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.955446 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.955687 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:22.955975 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.957117 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.958309 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.958993 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.959069 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.959126 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.959324 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.959373 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260808 02:25:22.959960 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.955384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.30.160.2:38365)
0808 02:25:22.955612 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.955616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.955670 (+ 54us) server_negotiation.cc:407] Connection header received
0808 02:25:22.955757 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.955759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.955777 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.955884 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:22.956130 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.957175 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.958634 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.959256 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.959326 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.959385 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.959775 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.959834 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260808 02:25:22.959967 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.955107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.30.160.1:42033)
0808 02:25:22.955836 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.955841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.955852 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:25:22.955917 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.955920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.955939 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.956031 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:22.956156 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.956919 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.958585 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.959337 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.959699 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.959774 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.959803 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.959855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":79,"threads_started":1}
I20260808 02:25:22.959720 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.954989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42033 (local address 127.0.0.1:58072)
0808 02:25:22.955275 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.955286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.955336 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.956028 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.956029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.956031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:22.956078 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.956080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.957036 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.957037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.958467 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.958470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.959439 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.959440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.959508 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.959514 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:25:22.959549 (+ 35us) client_negotiation.cc:240] Negotiation successful
0808 02:25:22.959592 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":22,"thread_start_us":89,"threads_started":1}
I20260808 02:25:22.959726 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.955166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38365 (local address 127.0.0.1:35248)
0808 02:25:22.955611 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.955623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.955683 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.955990 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.955991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.955993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:22.956046 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.956048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.957282 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.957283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.958558 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.958560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.959423 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.959424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.959492 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.959496 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:22.959558 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:25:22.959622 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260808 02:25:22.960525 15390 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca915900949416ebf6cd8e455567eb8 (DEFAULT_TABLE table=test-table [id=4bad8bdb0b7b45e1be60aa3576749181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:22.960918 15390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca915900949416ebf6cd8e455567eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:22.961081 15246 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca915900949416ebf6cd8e455567eb8 (DEFAULT_TABLE table=test-table [id=4bad8bdb0b7b45e1be60aa3576749181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:22.961402 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca915900949416ebf6cd8e455567eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:22.963997 15479 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Bootstrap starting.
I20260808 02:25:22.964368 15102 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca915900949416ebf6cd8e455567eb8 (DEFAULT_TABLE table=test-table [id=4bad8bdb0b7b45e1be60aa3576749181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:22.964558 15480 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Bootstrap starting.
I20260808 02:25:22.964709 15102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca915900949416ebf6cd8e455567eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:22.964957 15479 tablet_bootstrap.cc:654] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:22.965201 15480 tablet_bootstrap.cc:654] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:22.965538 15479 log.cc:824] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:22.965796 15480 log.cc:824] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:22.966462 15479 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: No bootstrap required, opened a new log
I20260808 02:25:22.966543 15479 ts_tablet_manager.cc:1397] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:25:22.966634 15480 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: No bootstrap required, opened a new log
I20260808 02:25:22.966713 15480 ts_tablet_manager.cc:1397] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:22.968212 15483 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap starting.
I20260808 02:25:22.968398 15480 raft_consensus.cc:348] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:22.968611 15480 raft_consensus.cc:374] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:22.968660 15480 raft_consensus.cc:729] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacf84c4da3c4ed19f2db86fd8aed978, State: Initialized, Role: FOLLOWER
I20260808 02:25:22.968683 15479 raft_consensus.cc:348] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:22.968871 15479 raft_consensus.cc:374] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:22.968819 15480 consensus_queue.cc:260] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:22.968917 15479 raft_consensus.cc:729] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d6ce5e04a1485c8482a50a26d34aff, State: Initialized, Role: FOLLOWER
I20260808 02:25:22.968923 15483 tablet_bootstrap.cc:654] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:22.969043 15479 consensus_queue.cc:260] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:22.969193 15480 ts_tablet_manager.cc:1428] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:25:22.969272 15317 heartbeater.cc:503] Master 127.30.160.62:39831 was elected leader, sending a full tablet report...
I20260808 02:25:22.969323 15479 ts_tablet_manager.cc:1428] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 02:25:22.969419 15461 heartbeater.cc:503] Master 127.30.160.62:39831 was elected leader, sending a full tablet report...
I20260808 02:25:22.969518 15483 log.cc:824] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:22.970681 15483 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: No bootstrap required, opened a new log
I20260808 02:25:22.970755 15483 ts_tablet_manager.cc:1397] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:25:22.972214 15483 raft_consensus.cc:348] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:22.972360 15483 raft_consensus.cc:374] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:22.972409 15483 raft_consensus.cc:729] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce410e069984f8ba0242f3d0fc2efe0, State: Initialized, Role: FOLLOWER
I20260808 02:25:22.972522 15483 consensus_queue.cc:260] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:22.972815 15483 ts_tablet_manager.cc:1428] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:22.972822 15174 heartbeater.cc:503] Master 127.30.160.62:39831 was elected leader, sending a full tablet report...
W20260808 02:25:23.022258 15318 tablet.cc:2369] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:23.088292 15484 raft_consensus.cc:482] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:23.088426 15484 raft_consensus.cc:504] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:23.088967 15484 leader_election.cc:288] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
I20260808 02:25:23.092526 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.089203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:32971 (local address 127.30.160.1:42033)
0808 02:25:23.089285 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:23.089287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:23.089301 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:25:23.089335 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:23.089336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:23.089348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:23.089418 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:23.089589 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.090372 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.091633 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.092145 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.092204 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.092211 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:23.092399 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 02:25:23.092446 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:25:23.092547 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.089295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:40773 (local address 127.30.160.3:42073)
0808 02:25:23.089359 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:23.089360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:23.089449 (+ 89us) server_negotiation.cc:407] Connection header received
0808 02:25:23.089530 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:23.089531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:23.089539 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:23.089616 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:23.089822 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.090591 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.091701 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.092163 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.092255 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.092265 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:23.092442 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:25:23.092487 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:25:23.092537 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.089146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42033 (local address 127.30.160.2:32971)
0808 02:25:23.089218 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:23.089231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:23.089279 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:23.089429 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:23.089430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:23.089432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:23.089510 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.089512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.090473 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.090474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.091545 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.091548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.092230 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.092231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.092319 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.092322 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:25:23.092375 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:25:23.092430 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":31}
I20260808 02:25:23.092562 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.089167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:42073 (local address 127.30.160.2:40773)
0808 02:25:23.089391 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:23.089401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:23.089471 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:23.089638 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:23.089639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:23.089640 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:23.089739 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.089741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.090656 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.090658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.091558 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.091561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.092271 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.092272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.092345 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.092349 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:23.092425 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:25:23.092461 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":115,"threads_started":1}
I20260808 02:25:23.092890 15411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca915900949416ebf6cd8e455567eb8" candidate_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" is_pre_election: true
I20260808 02:25:23.092988 15126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca915900949416ebf6cd8e455567eb8" candidate_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" is_pre_election: true
I20260808 02:25:23.093070 15411 raft_consensus.cc:2454] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacf84c4da3c4ed19f2db86fd8aed978 in term 0.
I20260808 02:25:23.093168 15126 raft_consensus.cc:2454] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacf84c4da3c4ed19f2db86fd8aed978 in term 0.
I20260808 02:25:23.093304 15197 leader_election.cc:302] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff; no voters:
I20260808 02:25:23.093441 15484 raft_consensus.cc:2781] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:23.093503 15484 raft_consensus.cc:482] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:23.093538 15484 raft_consensus.cc:3037] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:23.094274 15484 raft_consensus.cc:504] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:23.094419 15484 leader_election.cc:288] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [CANDIDATE]: Term 1 election: Requested vote from peers 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
I20260808 02:25:23.094564 15126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca915900949416ebf6cd8e455567eb8" candidate_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
I20260808 02:25:23.094583 15411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca915900949416ebf6cd8e455567eb8" candidate_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d6ce5e04a1485c8482a50a26d34aff"
I20260808 02:25:23.094643 15126 raft_consensus.cc:3037] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:23.094650 15411 raft_consensus.cc:3037] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:23.095558 15411 raft_consensus.cc:2454] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cacf84c4da3c4ed19f2db86fd8aed978 in term 1.
I20260808 02:25:23.095558 15126 raft_consensus.cc:2454] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cacf84c4da3c4ed19f2db86fd8aed978 in term 1.
I20260808 02:25:23.095752 15197 leader_election.cc:302] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff; no voters:
I20260808 02:25:23.095865 15484 raft_consensus.cc:2781] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:23.096035 15484 raft_consensus.cc:686] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 LEADER]: Becoming Leader. State: Replica: cacf84c4da3c4ed19f2db86fd8aed978, State: Running, Role: LEADER
I20260808 02:25:23.096161 15484 consensus_queue.cc:237] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:23.097066 14971 catalog_manager.cc:5495] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 reported cstate change: term changed from 0 to 1, leader changed from <none> to cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2). New cstate: current_term: 1 leader_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } health_report { overall_health: HEALTHY } } }
I20260808 02:25:23.102238 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.100030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.30.160.2:38365)
0808 02:25:23.100078 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:23.100080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:23.100091 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:25:23.100167 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:23.100168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:23.100182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:23.100237 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:23.100346 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.100756 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.101423 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.101592 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.101639 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.101767 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:23.101880 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:23.102045 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:23.102089 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 02:25:23.102124 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:25:23.103852 15126 raft_consensus.cc:1261] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Refusing update from remote peer cacf84c4da3c4ed19f2db86fd8aed978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:23.103973 15411 raft_consensus.cc:1261] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Refusing update from remote peer cacf84c4da3c4ed19f2db86fd8aed978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:23.104211 15484 consensus_queue.cc:1035] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:23.104343 15489 consensus_queue.cc:1035] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:23.108767 15495 mvcc.cc:205] Tried to move back new op lower bound from 7316094661031059456 to 7316094661002522624. Current Snapshot: MvccSnapshot[applied={T|T < 7316094661031059456 or (T in {7316094661031059456})}]
I20260808 02:25:23.109262 14971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:23.109361 14971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:23.110132 14971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Sending DeleteTablet for 3 replicas of tablet 6ca915900949416ebf6cd8e455567eb8
I20260808 02:25:23.110502 15102 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca915900949416ebf6cd8e455567eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:23 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:58072
I20260808 02:25:23.110536 15390 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca915900949416ebf6cd8e455567eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:23 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:35598
I20260808 02:25:23.110538 15246 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca915900949416ebf6cd8e455567eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:23 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:35248
I20260808 02:25:23.110723 15503 tablet_replica.cc:330] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: stopping tablet replica
I20260808 02:25:23.110847 15503 raft_consensus.cc:2229] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:23.110854 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15037
I20260808 02:25:23.110956 15503 raft_consensus.cc:2258] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:23.110977 15504 tablet_replica.cc:330] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: stopping tablet replica
I20260808 02:25:23.111268 15504 raft_consensus.cc:2229] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:23.111488 15503 ts_tablet_manager.cc:1905] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:23.112006 15505 tablet_replica.cc:330] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: stopping tablet replica
I20260808 02:25:23.112164 15505 raft_consensus.cc:2229] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:23.112306 15505 raft_consensus.cc:2258] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:23.112726 15505 ts_tablet_manager.cc:1905] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:23.113440 15503 ts_tablet_manager.cc:1918] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:23.113519 15503 log.cc:1196] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/6ca915900949416ebf6cd8e455567eb8
I20260808 02:25:23.113859 15503 ts_tablet_manager.cc:1939] T 6ca915900949416ebf6cd8e455567eb8 P e1d6ce5e04a1485c8482a50a26d34aff: Deleting consensus metadata
I20260808 02:25:23.114382 14954 catalog_manager.cc:4841] TS e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): tablet 6ca915900949416ebf6cd8e455567eb8 (table test-table [id=4bad8bdb0b7b45e1be60aa3576749181]) successfully deleted
I20260808 02:25:23.114938 15505 ts_tablet_manager.cc:1918] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:23.115024 15505 log.cc:1196] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/6ca915900949416ebf6cd8e455567eb8
I20260808 02:25:23.115279 15505 ts_tablet_manager.cc:1939] T 6ca915900949416ebf6cd8e455567eb8 P cacf84c4da3c4ed19f2db86fd8aed978: Deleting consensus metadata
I20260808 02:25:23.115624 14954 catalog_manager.cc:4841] TS cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): tablet 6ca915900949416ebf6cd8e455567eb8 (table test-table [id=4bad8bdb0b7b45e1be60aa3576749181]) successfully deleted
W20260808 02:25:23.124795 14957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:42033 (error 108)
I20260808 02:25:23.125453 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.125327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42033 (local address 127.0.0.1:58084)
0808 02:25:23.125381 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:23.125426 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:25:23.125443 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15179
W20260808 02:25:23.126659 14957 catalog_manager.cc:4584] TS 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ca915900949416ebf6cd8e455567eb8: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42033: connect: Connection refused (error 111)
I20260808 02:25:23.132472 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15322
I20260808 02:25:23.137624 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 14938
I20260808 02:25:43.143882 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39831
--webserver_interface=127.30.160.62
--webserver_port=44709
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:43.226610 15507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:43.226820 15507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:43.226843 15507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:43.228040 15507 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:25:43.228082 15507 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:25:43.228096 15507 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:25:43.228108 15507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:43.228120 15507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:43.228134 15507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:43.228147 15507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:43.228204 15507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:43.229660 15507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:43.229916 15507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:43.230165 15507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:43.239720 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.239745 15517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.239745 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:43.239730 15507 server_base.cc:1034] running on GCE node
I20260808 02:25:43.240192 15507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:43.240521 15507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:43.241660 15507 hybrid_clock.cc:648] HybridClock initialized: now 1786155943241644 us; error 32 us; skew 500 ppm
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:43.249208 15507 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:25:43.249697 15507 webserver.cc:460] Webserver started at http://127.30.160.62:44709/ using document root <none> and password file <none>
I20260808 02:25:43.249898 15507 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:43.249954 15507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:43.251319 15507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:43.251986 15525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.252158 15507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:43.252264 15507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "5575f29f77b340d8bc5852418061ebb6"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "38edd528208b3693a6a8c356b95b8ce4"
server_key_iv: "e45d53538de523321bd21a1d4d5c91a5"
server_key_version: "encryptionkey@0"
I20260808 02:25:43.252699 15507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:43.276490 15507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:43.278527 15507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:43.278721 15507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:43.283285 15507 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39831
I20260808 02:25:43.283329 15581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39831 every 8 connection(s)
I20260808 02:25:43.283806 15507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:43.285328 15582 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:25:43.285943 15582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap starting.
I20260808 02:25:43.290017 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15507
I20260808 02:25:43.290357 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:42033
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:39831
--webserver_port=44985
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:43.294582 15582 log.cc:824] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:43.296541 15582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:43.296845 15582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap complete.
I20260808 02:25:43.299110 15582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:43.299355 15582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5575f29f77b340d8bc5852418061ebb6, State: Initialized, Role: FOLLOWER
I20260808 02:25:43.299533 15582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:43.299610 15582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:43.299669 15582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:43.299717 15582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:25:43.300763 15582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:43.300891 15582 leader_election.cc:302] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5575f29f77b340d8bc5852418061ebb6; no voters:
I20260808 02:25:43.301074 15582 leader_election.cc:288] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:25:43.301156 15586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:25:43.301338 15582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:43.301385 15586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 LEADER]: Becoming Leader. State: Replica: 5575f29f77b340d8bc5852418061ebb6, State: Running, Role: LEADER
I20260808 02:25:43.301520 15586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:25:43.301888 15587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5575f29f77b340d8bc5852418061ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } } }
I20260808 02:25:43.301975 15587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:43.302110 15587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5575f29f77b340d8bc5852418061ebb6. Latest consensus state: current_term: 2 leader_uuid: "5575f29f77b340d8bc5852418061ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } } }
I20260808 02:25:43.302173 15587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:43.302527 15596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:43.303377 15596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:43.303484 15596 catalog_manager.cc:1259] Loaded cluster ID: 8ff0e8664cae4cb2aa335854c0ac1ba1
I20260808 02:25:43.303519 15596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:43.304934 15596 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:43.305133 15596 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Loaded TSK: 0
I20260808 02:25:43.320897 15596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Generated new TSK 1
I20260808 02:25:43.321004 15596 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:25:43.383178 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:43.383387 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:43.383412 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:43.384629 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:43.384673 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:43.384699 15584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:25:43.386202 15584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:42033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=44985
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:43.386446 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:43.386704 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:43.391746 15610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.391731 15611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:43.391976 15584 server_base.cc:1034] running on GCE node
W20260808 02:25:43.391721 15613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:43.392324 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:43.392604 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:43.393795 15584 hybrid_clock.cc:648] HybridClock initialized: now 1786155943393766 us; error 43 us; skew 500 ppm
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:43.396770 15584 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:25:43.397262 15584 webserver.cc:460] Webserver started at http://127.30.160.1:44985/ using document root <none> and password file <none>
I20260808 02:25:43.397516 15584 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:43.397603 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:43.399022 15584 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:43.399748 15622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.399938 15584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:43.400029 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "fc03c7c7dc37b49c0fdbb353aeccbb1c"
server_key_iv: "af5631dec0a8c8f05b7d43d1aa70fe64"
server_key_version: "encryptionkey@0"
I20260808 02:25:43.400363 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:43.409943 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:43.412282 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:43.412448 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:43.412676 15584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:43.413110 15629 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:25:43.414959 15584 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:25:43.415024 15584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 02:25:43.415067 15584 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:25:43.415731 15584 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:25:43.415791 15584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 02:25:43.415853 15629 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap starting.
I20260808 02:25:43.423166 15584 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42033
I20260808 02:25:43.423540 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:43.426507 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15584
I20260808 02:25:43.426815 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:38365
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:39831
--webserver_port=34679
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:43.427145 15629 log.cc:824] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:43.428900 15747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42033 every 8 connection(s)
I20260808 02:25:43.430227 15629 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:43.430531 15629 tablet_bootstrap.cc:492] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap complete.
I20260808 02:25:43.431859 15629 ts_tablet_manager.cc:1397] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260808 02:25:43.433194 15629 raft_consensus.cc:348] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:43.433519 15629 raft_consensus.cc:729] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce410e069984f8ba0242f3d0fc2efe0, State: Initialized, Role: FOLLOWER
I20260808 02:25:43.433755 15629 consensus_queue.cc:260] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } }
I20260808 02:25:43.434053 15629 ts_tablet_manager.cc:1428] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:43.441538 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.424369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:45011 (local address 127.30.160.62:39831)
0808 02:25:43.429216 (+ 4847us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.429227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.429272 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:25:43.429414 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.429421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.429480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.429628 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:43.430534 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.432058 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.434094 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.434830 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.434949 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.438068 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:43.438071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:43.438071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:43.438082 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:43.439771 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.440313 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.440316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.440317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.440404 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.440675 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.440676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.440676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.440905 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:43.440968 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.441298 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.441401 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4788,"thread_start_us":88,"threads_started":1}
I20260808 02:25:43.441601 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.424303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:45011)
0808 02:25:43.427435 (+ 3132us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.427478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.429101 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.429789 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.429794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.430144 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:43.430400 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.430407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.432181 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.432182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.433788 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.433792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.435055 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.435056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.435209 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.435940 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:43.435951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:43.437947 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:43.439876 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.439885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.439888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.440218 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.440473 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.440473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.440474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.440606 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.440975 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:43.440981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:43.441132 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.441233 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.441394 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3089,"thread_start_us":110,"threads_started":1}
I20260808 02:25:43.442413 15748 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39831
I20260808 02:25:43.442516 15748 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:43.442693 15748 heartbeater.cc:511] Master 127.30.160.62:39831 requested a full tablet report, sending...
I20260808 02:25:43.443441 15545 ts_manager.cc:194] Registered new tserver with Master: 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033)
I20260808 02:25:43.444604 15545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:45011
I20260808 02:25:43.445538 15748 heartbeater.cc:503] Master 127.30.160.62:39831 was elected leader, sending a full tablet report...
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.1@KRBTEST.COM
I20260808 02:25:43.452486 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.444055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42033 (local address 127.0.0.1:51468)
0808 02:25:43.444425 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.444459 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.444538 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.444765 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.444772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.445089 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:43.445143 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.445145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.446194 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.446195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.447130 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.447133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.447558 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.447559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.447639 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.448117 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:43.448122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:43.449494 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:43.451252 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.451260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.451262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.451464 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.451697 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.451697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.451698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.451760 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.452104 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:43.452111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:43.452162 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.452263 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.452390 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":102,"threads_started":1}
I20260808 02:25:43.452486 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.444262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.30.160.1:42033)
0808 02:25:43.444432 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.444437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.444535 (+ 98us) server_negotiation.cc:407] Connection header received
0808 02:25:43.444610 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.444612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.444626 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.444698 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:43.445227 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.446114 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.447212 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.447479 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.447531 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.449585 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:43.449593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:43.449593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:43.449600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:43.451143 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.451560 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.451562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.451563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.451621 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.451827 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.451828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.451828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.452029 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:43.452124 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.452302 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.452390 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260808 02:25:43.452844 15675 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca915900949416ebf6cd8e455567eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:23 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:51468
I20260808 02:25:43.453348 15757 tablet_replica.cc:330] stopping tablet replica
I20260808 02:25:43.453464 15757 raft_consensus.cc:2229] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:43.453532 15757 raft_consensus.cc:2258] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:43.453858 15757 ts_tablet_manager.cc:1905] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:43.456506 15757 ts_tablet_manager.cc:1918] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:43.456588 15757 log.cc:1196] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal/wals/6ca915900949416ebf6cd8e455567eb8
I20260808 02:25:43.456831 15757 ts_tablet_manager.cc:1939] T 6ca915900949416ebf6cd8e455567eb8 P 3ce410e069984f8ba0242f3d0fc2efe0: Deleting consensus metadata
I20260808 02:25:43.457366 15529 catalog_manager.cc:4841] TS 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033): tablet 6ca915900949416ebf6cd8e455567eb8 (table test-table [id=4bad8bdb0b7b45e1be60aa3576749181]) successfully deleted
W20260808 02:25:43.519886 15753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:43.520102 15753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:43.520124 15753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:43.521261 15753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:43.521303 15753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:43.521330 15753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:25:43.522778 15753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:38365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=34679
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:43.523018 15753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:43.523283 15753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:43.528348 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.528362 15763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.528391 15764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:43.528393 15753 server_base.cc:1034] running on GCE node
I20260808 02:25:43.528817 15753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:43.529076 15753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:43.530243 15753 hybrid_clock.cc:648] HybridClock initialized: now 1786155943530230 us; error 24 us; skew 500 ppm
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:43.532768 15753 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:25:43.533200 15753 webserver.cc:460] Webserver started at http://127.30.160.2:34679/ using document root <none> and password file <none>
I20260808 02:25:43.533434 15753 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:43.533524 15753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:43.534864 15753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:43.535398 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.535589 15753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:25:43.535660 15753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "cacf84c4da3c4ed19f2db86fd8aed978"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "eb98630dc702a32fd7d9d1e23b029a68"
server_key_iv: "30f949b26d4121b9d4bc4fc7957e73f6"
server_key_version: "encryptionkey@0"
I20260808 02:25:43.536031 15753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:43.557884 15753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:43.559767 15753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:43.559877 15753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:43.560088 15753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:43.560424 15753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:43.560456 15753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.560516 15753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:43.560549 15753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.566614 15753 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38365
I20260808 02:25:43.566672 15893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38365 every 8 connection(s)
I20260808 02:25:43.566995 15753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:43.574270 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15753
I20260808 02:25:43.574572 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:42073
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:39831
--webserver_port=44031
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:43.576490 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.567706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:34085 (local address 127.30.160.62:39831)
0808 02:25:43.567772 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.567774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.568013 (+ 239us) server_negotiation.cc:407] Connection header received
0808 02:25:43.568545 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.568547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.568560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.568618 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:43.569558 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.570063 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.571050 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.571268 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.571378 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.573465 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:43.573466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:43.573466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:43.573473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:43.574966 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.575345 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.575346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.575347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.575411 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.575643 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.575643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.575644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.575790 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:43.575860 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.576152 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.576249 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:25:43.576532 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.567711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:34085)
0808 02:25:43.567868 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.567918 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.568437 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.568691 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.568694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.569260 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:43.569450 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.569455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.570169 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.570170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.570950 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.570952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.571390 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.571391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.571526 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.572060 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:43.572068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:43.573377 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:43.575042 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.575049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.575051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.575260 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.575486 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.575486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.575487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.575580 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.575853 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:43.575857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:43.576024 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.576108 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.576256 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260808 02:25:43.577510 15894 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39831
I20260808 02:25:43.577603 15894 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:43.577797 15894 heartbeater.cc:511] Master 127.30.160.62:39831 requested a full tablet report, sending...
I20260808 02:25:43.578169 15545 ts_manager.cc:194] Registered new tserver with Master: cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365)
I20260808 02:25:43.578555 15545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:34085
W20260808 02:25:43.658782 15897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:43.659026 15897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:43.659065 15897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:43.660252 15897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:43.660310 15897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:43.660372 15897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:25:43.661846 15897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=44031
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:25:43.662096 15897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:43.662379 15897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:43.667284 15902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.667299 15903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:43.667320 15905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:43.667470 15897 server_base.cc:1034] running on GCE node
I20260808 02:25:43.667999 15897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:43.668246 15897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:43.669410 15897 hybrid_clock.cc:648] HybridClock initialized: now 1786155943669410 us; error 42 us; skew 500 ppm
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:43.671988 15897 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:25:43.672371 15897 webserver.cc:460] Webserver started at http://127.30.160.3:44031/ using document root <none> and password file <none>
I20260808 02:25:43.672542 15897 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:43.672595 15897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:43.673856 15897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:43.674378 15912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.674546 15897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:25:43.674634 15897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "e1d6ce5e04a1485c8482a50a26d34aff"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "b6c59d4b4deceb6cde242d80e95fd653"
server_key_iv: "d4934549a44f8d81b04b88ded67b5abd"
server_key_version: "encryptionkey@0"
I20260808 02:25:43.674942 15897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:43.698791 15897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:43.700855 15897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:43.701009 15897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:43.701346 15897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:43.701649 15897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:43.701681 15897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.701735 15897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:43.701761 15897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:43.707624 15897 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:42073
I20260808 02:25:43.707711 16036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:42073 every 8 connection(s)
I20260808 02:25:43.708055 15897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:43.711514 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 15897
Aug 08 02:25:43 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155943, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:25:43.718051 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.708795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:55845 (local address 127.30.160.62:39831)
0808 02:25:43.708868 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.708871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.709120 (+ 249us) server_negotiation.cc:407] Connection header received
0808 02:25:43.709572 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.709573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.709583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.709636 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:43.710710 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.711200 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.712367 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.712605 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.712656 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.715111 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:43.715112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:43.715112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:43.715117 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:43.716727 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.717106 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.717107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.717108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.717172 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.717437 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.717438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.717438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.717551 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:43.717627 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.717926 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.717988 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:25:43.718294 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.708794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:55845)
0808 02:25:43.708993 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.709030 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.709486 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.709698 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.709701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.710273 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:43.710496 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.710501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.711293 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.711294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.712289 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.712291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.712688 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.712689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.712854 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.713427 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:43.713436 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:43.715018 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:43.716792 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.716798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.716800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.717017 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.717244 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:43.717245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:43.717246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:43.717357 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:43.717618 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:43.717625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:43.717758 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.717879 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.718047 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260808 02:25:43.718766 16038 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39831
I20260808 02:25:43.718878 16038 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:43.719074 16038 heartbeater.cc:511] Master 127.30.160.62:39831 requested a full tablet report, sending...
I20260808 02:25:43.719148 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.711988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.30.160.62:39831)
0808 02:25:43.712135 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.712137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.712234 (+ 97us) server_negotiation.cc:407] Connection header received
0808 02:25:43.712310 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.712311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.712319 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.712410 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:43.713105 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.713676 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.714554 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.714828 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.714885 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.716324 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:43.716325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:43.716325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:43.716334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:43.717764 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.718149 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.718150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.718151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.718206 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.718572 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.718573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.718573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.718716 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:43.718885 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.718973 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.719055 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260808 02:25:43.719425 15542 ts_manager.cc:194] Registered new tserver with Master: e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
I20260808 02:25:43.719841 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:25:43.719858 15542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:55845
I20260808 02:25:43.722623 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.720224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.30.160.62:39831)
0808 02:25:43.720262 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.720264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.720701 (+ 437us) server_negotiation.cc:407] Connection header received
0808 02:25:43.720758 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.720759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.720785 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.720842 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:43.720907 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.721350 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.722060 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.722234 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.722281 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.722415 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:43.722451 (+ 36us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 02:25:43.722561 (+ 110us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33644: token expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:25:43.722678 16041 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33644: token expired
W20260808 02:25:43.722864 15467 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.160.62:39831)
I20260808 02:25:43.728106 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.723065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.30.160.62:39831)
0808 02:25:43.723097 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.723099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.723142 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:25:43.723186 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.723187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.723193 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.723234 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:43.723632 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.724107 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.724734 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.724976 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.725029 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.725995 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:43.725996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:43.725996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:43.726002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:43.727152 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.727445 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.727446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.727446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.727493 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:43.727653 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:43.727653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:43.727654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:43.727751 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:43.727820 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.727978 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.728035 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 02:25:43.729195 15542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:43.730664 15542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:43.732564 15675 tablet_service.cc:1467] Processing CreateTablet for tablet 01434be8a5d54d5088a1b76ffdd5ce3a (DEFAULT_TABLE table=test-table [id=c1dd4ef5a64741c6aa2ea46dbeed7835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:43.732779 15675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01434be8a5d54d5088a1b76ffdd5ce3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:43.734380 15629 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap starting.
I20260808 02:25:43.734719 15629 tablet_bootstrap.cc:654] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:43.735333 15629 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: No bootstrap required, opened a new log
I20260808 02:25:43.735381 15629 ts_tablet_manager.cc:1397] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:25:43.735754 15629 raft_consensus.cc:348] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.735852 15629 raft_consensus.cc:374] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:43.735946 15629 raft_consensus.cc:729] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce410e069984f8ba0242f3d0fc2efe0, State: Initialized, Role: FOLLOWER
I20260808 02:25:43.736001 15629 consensus_queue.cc:260] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.736106 15629 ts_tablet_manager.cc:1428] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:25:43.736722 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.732341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38365 (local address 127.0.0.1:33592)
0808 02:25:43.732576 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.732586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.732628 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.732914 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.732915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.732918 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:43.732959 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.732961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.734197 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.734198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.735451 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.735455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.736459 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.736460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.736536 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.736554 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.736610 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.736656 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":65}
I20260808 02:25:43.736757 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.732498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.30.160.2:38365)
0808 02:25:43.732691 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.732696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.732707 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:25:43.732748 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.732750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.732766 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.732906 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:43.733022 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.734084 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.735560 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.736376 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.736453 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.736506 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.736630 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.736672 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260808 02:25:43.737368 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.732740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.30.160.3:42073)
0808 02:25:43.732937 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.732942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.732952 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:25:43.732984 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.732986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.733002 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.733082 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:43.733194 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.734069 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.735772 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.736769 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.737130 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.737177 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.737281 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.737310 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260808 02:25:43.737366 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.732434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:42073 (local address 127.0.0.1:52386)
0808 02:25:43.732768 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.732778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.732826 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.733079 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.733080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.733082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:43.733140 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.733141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.734166 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.734167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.735691 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.735694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.737142 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.737143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.737212 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.737216 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.737271 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.737304 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":49,"thread_start_us":100,"threads_started":1}
I20260808 02:25:43.737480 15823 tablet_service.cc:1467] Processing CreateTablet for tablet 01434be8a5d54d5088a1b76ffdd5ce3a (DEFAULT_TABLE table=test-table [id=c1dd4ef5a64741c6aa2ea46dbeed7835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:43.737816 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01434be8a5d54d5088a1b76ffdd5ce3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:43.740731 16048 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Bootstrap starting.
I20260808 02:25:43.741281 15964 tablet_service.cc:1467] Processing CreateTablet for tablet 01434be8a5d54d5088a1b76ffdd5ce3a (DEFAULT_TABLE table=test-table [id=c1dd4ef5a64741c6aa2ea46dbeed7835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:43.741374 16048 tablet_bootstrap.cc:654] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:43.741585 15964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01434be8a5d54d5088a1b76ffdd5ce3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:43.741986 16048 log.cc:824] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:43.742722 16048 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: No bootstrap required, opened a new log
I20260808 02:25:43.742806 16048 ts_tablet_manager.cc:1397] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:43.744511 16048 raft_consensus.cc:348] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.744683 16048 raft_consensus.cc:374] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:43.744727 16048 raft_consensus.cc:729] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacf84c4da3c4ed19f2db86fd8aed978, State: Initialized, Role: FOLLOWER
I20260808 02:25:43.744874 16048 consensus_queue.cc:260] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.744917 16050 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Bootstrap starting.
I20260808 02:25:43.745158 16048 ts_tablet_manager.cc:1428] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 02:25:43.745325 15894 heartbeater.cc:503] Master 127.30.160.62:39831 was elected leader, sending a full tablet report...
I20260808 02:25:43.745699 16050 tablet_bootstrap.cc:654] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:43.746263 16050 log.cc:824] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:43.747219 16050 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: No bootstrap required, opened a new log
I20260808 02:25:43.747300 16050 ts_tablet_manager.cc:1397] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:43.749102 16050 raft_consensus.cc:348] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.749230 16050 raft_consensus.cc:374] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:43.749250 16050 raft_consensus.cc:729] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d6ce5e04a1485c8482a50a26d34aff, State: Initialized, Role: FOLLOWER
I20260808 02:25:43.749398 16050 consensus_queue.cc:260] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.749660 16050 ts_tablet_manager.cc:1428] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:25:43.749814 16038 heartbeater.cc:503] Master 127.30.160.62:39831 was elected leader, sending a full tablet report...
I20260808 02:25:43.778178 16053 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:43.778366 16053 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.778725 16053 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033)
I20260808 02:25:43.781867 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.779010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:44859 (local address 127.30.160.1:42033)
0808 02:25:43.779057 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.779059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.779072 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:25:43.779095 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.779096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.779117 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.779207 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:43.779378 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.780035 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.781003 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.781534 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.781607 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.781683 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.781761 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.781805 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:25:43.781863 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.778892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42033 (local address 127.30.160.3:44859)
0808 02:25:43.778962 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.778976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.779018 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.779214 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.779215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.779217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:43.779298 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.779299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.780118 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.780119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.780899 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.780901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.781604 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.781605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.781687 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.781691 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.781750 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.781788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":27}
I20260808 02:25:43.781958 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.778971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:52857 (local address 127.30.160.2:38365)
0808 02:25:43.779022 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.779024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.779142 (+ 118us) server_negotiation.cc:407] Connection header received
0808 02:25:43.779174 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.779175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.779185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.779262 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:43.779424 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.779957 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.781006 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.781562 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.781635 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.781646 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.781855 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.781891 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:25:43.781980 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.778908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38365 (local address 127.30.160.3:52857)
0808 02:25:43.779062 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.779070 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:43.779119 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:43.779266 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:43.779267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:43.779268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:43.779354 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.779356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.780052 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.780053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.780895 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:43.780898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.781683 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.781684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:43.781758 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.781762 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:25:43.781850 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:25:43.781890 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":34,"thread_start_us":96,"threads_started":1}
I20260808 02:25:43.782178 15694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01434be8a5d54d5088a1b76ffdd5ce3a" candidate_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" is_pre_election: true
I20260808 02:25:43.782378 15694 raft_consensus.cc:2454] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d6ce5e04a1485c8482a50a26d34aff in term 0.
I20260808 02:25:43.782387 15846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01434be8a5d54d5088a1b76ffdd5ce3a" candidate_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" is_pre_election: true
I20260808 02:25:43.782532 15846 raft_consensus.cc:2454] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d6ce5e04a1485c8482a50a26d34aff in term 0.
I20260808 02:25:43.782615 15917 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce410e069984f8ba0242f3d0fc2efe0, e1d6ce5e04a1485c8482a50a26d34aff; no voters:
I20260808 02:25:43.782791 16053 raft_consensus.cc:2781] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:43.782840 16053 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:43.782855 16053 raft_consensus.cc:3037] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:43.783600 16053 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.783735 16053 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [CANDIDATE]: Term 1 election: Requested vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033)
I20260808 02:25:43.783891 15846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01434be8a5d54d5088a1b76ffdd5ce3a" candidate_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacf84c4da3c4ed19f2db86fd8aed978"
I20260808 02:25:43.783989 15846 raft_consensus.cc:3037] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:43.784698 15846 raft_consensus.cc:2454] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d6ce5e04a1485c8482a50a26d34aff in term 1.
I20260808 02:25:43.784814 15694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01434be8a5d54d5088a1b76ffdd5ce3a" candidate_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
I20260808 02:25:43.784888 15913 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff; no voters:
I20260808 02:25:43.784897 15694 raft_consensus.cc:3037] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:43.785562 15694 raft_consensus.cc:2454] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d6ce5e04a1485c8482a50a26d34aff in term 1.
I20260808 02:25:43.785645 16053 raft_consensus.cc:2781] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:43.785827 16053 raft_consensus.cc:686] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 1 LEADER]: Becoming Leader. State: Replica: e1d6ce5e04a1485c8482a50a26d34aff, State: Running, Role: LEADER
I20260808 02:25:43.785941 16053 consensus_queue.cc:237] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:25:43.786811 15542 catalog_manager.cc:5495] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff reported cstate change: term changed from 0 to 1, leader changed from <none> to e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3). New cstate: current_term: 1 leader_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:43.797276 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.795201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.30.160.3:42073)
0808 02:25:43.795244 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:43.795246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:43.795254 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:25:43.795297 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:43.795298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:43.795318 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:43.795375 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:43.795471 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.795970 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.796663 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:43.796830 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:43.796876 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:43.796984 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:43.797065 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:43.797116 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:43.797145 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 02:25:43.797159 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:25:43.798858 15846 raft_consensus.cc:1261] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Refusing update from remote peer e1d6ce5e04a1485c8482a50a26d34aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:43.798857 15694 raft_consensus.cc:1261] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Refusing update from remote peer e1d6ce5e04a1485c8482a50a26d34aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:43.799211 16055 consensus_queue.cc:1035] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:43.799355 16053 consensus_queue.cc:1035] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:43.803654 15542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:43.803750 15542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:43.804553 15542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Sending DeleteTablet for 3 replicas of tablet 01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:25:43.804849 15823 tablet_service.cc:1514] Processing DeleteTablet for tablet 01434be8a5d54d5088a1b76ffdd5ce3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:43 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:33592
I20260808 02:25:43.804896 15964 tablet_service.cc:1514] Processing DeleteTablet for tablet 01434be8a5d54d5088a1b76ffdd5ce3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:43 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:52386
I20260808 02:25:43.804935 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15584
I20260808 02:25:43.805886 16070 tablet_replica.cc:330] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: stopping tablet replica
I20260808 02:25:43.805992 16070 raft_consensus.cc:2229] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:43.806094 16070 raft_consensus.cc:2258] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:43.806551 16070 ts_tablet_manager.cc:1905] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:43.808454 16070 ts_tablet_manager.cc:1918] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:43.808530 16070 log.cc:1196] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:25:43.808837 16070 ts_tablet_manager.cc:1939] T 01434be8a5d54d5088a1b76ffdd5ce3a P cacf84c4da3c4ed19f2db86fd8aed978: Deleting consensus metadata
I20260808 02:25:43.809237 15526 catalog_manager.cc:4841] TS cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): tablet 01434be8a5d54d5088a1b76ffdd5ce3a (table test-table [id=c1dd4ef5a64741c6aa2ea46dbeed7835]) successfully deleted
I20260808 02:25:43.812036 16071 tablet_replica.cc:330] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: stopping tablet replica
I20260808 02:25:43.812151 16071 raft_consensus.cc:2229] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:43.812259 16071 raft_consensus.cc:2258] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:43.812724 16071 ts_tablet_manager.cc:1905] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:25:43.813470 15529 connection.cc:535] client connection to 127.30.160.1:42033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 02:25:43.813587 15529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:25:43.813863 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15753
I20260808 02:25:43.814862 16071 ts_tablet_manager.cc:1918] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:43.814940 16071 log.cc:1196] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:25:43.815225 16071 ts_tablet_manager.cc:1939] T 01434be8a5d54d5088a1b76ffdd5ce3a P e1d6ce5e04a1485c8482a50a26d34aff: Deleting consensus metadata
I20260808 02:25:43.815347 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:43.814118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:42033 (local address 127.0.0.1:51470)
0808 02:25:43.815256 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:43.815307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1099}
W20260808 02:25:43.815469 15529 catalog_manager.cc:4584] TS 3ce410e069984f8ba0242f3d0fc2efe0 (127.30.160.1:42033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01434be8a5d54d5088a1b76ffdd5ce3a: Network error: Client connection negotiation failed: client connection to 127.30.160.1:42033: connect: Connection refused (error 111)
I20260808 02:25:43.816095 15526 catalog_manager.cc:4841] TS e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): tablet 01434be8a5d54d5088a1b76ffdd5ce3a (table test-table [id=c1dd4ef5a64741c6aa2ea46dbeed7835]) successfully deleted
I20260808 02:25:43.820508 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15897
I20260808 02:25:43.826419 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 15507
I20260808 02:26:03.832865 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39831
--webserver_interface=127.30.160.62
--webserver_port=44709
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:03.917676 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:03.917865 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:03.917886 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:03.918931 16073 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:26:03.918962 16073 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:26:03.918975 16073 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:26:03.918987 16073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:03.918999 16073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:03.919013 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:03.919024 16073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:03.919039 16073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:03.920439 16073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=44709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:03.920619 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:03.920778 16073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:03.925488 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:03.925495 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:03.925524 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:03.925815 16073 server_base.cc:1034] running on GCE node
I20260808 02:26:03.926033 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:03.926254 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:03.927436 16073 hybrid_clock.cc:648] HybridClock initialized: now 1786155963927418 us; error 26 us; skew 500 ppm
Aug 08 02:26:03 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:03.933658 16073 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:03.934116 16073 webserver.cc:460] Webserver started at http://127.30.160.62:44709/ using document root <none> and password file <none>
I20260808 02:26:03.934365 16073 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:03.934434 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:03.935743 16073 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:03.936467 16090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:03.936640 16073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:03.936774 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "5575f29f77b340d8bc5852418061ebb6"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "38edd528208b3693a6a8c356b95b8ce4"
server_key_iv: "e45d53538de523321bd21a1d4d5c91a5"
server_key_version: "encryptionkey@0"
I20260808 02:26:03.937095 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:03.949623 16073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:03.951953 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:03.952112 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:03.955875 16073 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39831
I20260808 02:26:03.955967 16149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39831 every 8 connection(s)
I20260808 02:26:03.956346 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:03.957713 16150 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:26:03.958288 16150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap starting.
I20260808 02:26:03.959208 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16073
I20260808 02:26:03.959493 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:42033
--local_ip_for_outbound_sockets=127.30.160.1
--tserver_master_addrs=127.30.160.62:39831
--webserver_port=44985
--webserver_interface=127.30.160.1
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:26:03.967549 16150 log.cc:824] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:03.969776 16150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:26:03.970088 16150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Bootstrap complete.
I20260808 02:26:03.972391 16150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:26:03.972607 16150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5575f29f77b340d8bc5852418061ebb6, State: Initialized, Role: FOLLOWER
I20260808 02:26:03.972766 16150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:26:03.972822 16150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:03.972882 16150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:03.972934 16150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:26:03.973929 16150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:26:03.974038 16150 leader_election.cc:302] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5575f29f77b340d8bc5852418061ebb6; no voters:
I20260808 02:26:03.974236 16150 leader_election.cc:288] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:26:03.974272 16154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:26:03.974390 16154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [term 3 LEADER]: Becoming Leader. State: Replica: 5575f29f77b340d8bc5852418061ebb6, State: Running, Role: LEADER
I20260808 02:26:03.974481 16154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } }
I20260808 02:26:03.974515 16150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:03.974732 16154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5575f29f77b340d8bc5852418061ebb6. Latest consensus state: current_term: 3 leader_uuid: "5575f29f77b340d8bc5852418061ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } } }
I20260808 02:26:03.974807 16154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:03.974735 16155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5575f29f77b340d8bc5852418061ebb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5575f29f77b340d8bc5852418061ebb6" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39831 } } }
I20260808 02:26:03.974975 16155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:03.975018 16158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:03.975852 16158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:03.976048 16158 catalog_manager.cc:1259] Loaded cluster ID: 8ff0e8664cae4cb2aa335854c0ac1ba1
I20260808 02:26:03.976311 16158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:03.977138 16158 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:03.977334 16158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Loaded TSK: 0
I20260808 02:26:03.977391 16158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5575f29f77b340d8bc5852418061ebb6: Loaded TSK: 1
F20260808 02:26:03.977626 16158 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4ee3a459ad google::LogMessage::Fail() at ??:0
@ 0x7f4ee3a4888a google::LogMessage::SendToLog() at ??:0
@ 0x7f4ee3a454de google::LogMessage::Flush() at ??:0
@ 0x7f4ee3a46a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4ee61b2104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4ee61afd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4eeb9a31cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4eeb9a348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4eeb9a3594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4eeb969c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4eeb973ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4ee59130d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4ee5908954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4ee676c6db start_thread at ??:0
@ 0x7f4ee254361f clone at ??:0
W20260808 02:26:04.065243 16152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:04.065436 16152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:04.065460 16152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:04.066613 16152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:04.066656 16152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:04.066682 16152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:04.068172 16152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:42033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=44985
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:04.068436 16152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:04.068681 16152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:04.074728 16193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:04.074745 16192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:04.074930 16195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:04.075100 16152 server_base.cc:1034] running on GCE node
I20260808 02:26:04.075330 16152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:04.075584 16152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:04.076766 16152 hybrid_clock.cc:648] HybridClock initialized: now 1786155964076734 us; error 47 us; skew 500 ppm
Aug 08 02:26:04 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:04.079327 16152 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:26:04.079768 16152 webserver.cc:460] Webserver started at http://127.30.160.1:44985/ using document root <none> and password file <none>
I20260808 02:26:04.080040 16152 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:04.080112 16152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:04.081506 16152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:04.082245 16202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.082419 16152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:04.082520 16152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "3ce410e069984f8ba0242f3d0fc2efe0"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "fc03c7c7dc37b49c0fdbb353aeccbb1c"
server_key_iv: "af5631dec0a8c8f05b7d43d1aa70fe64"
server_key_version: "encryptionkey@0"
I20260808 02:26:04.082863 16152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:04.101949 16152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:04.104389 16152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:04.104553 16152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:04.104791 16152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:04.105240 16211 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:26:04.107191 16152 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:26:04.107252 16152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 02:26:04.107321 16152 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:26:04.107998 16152 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:26:04.108052 16152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:04.108115 16211 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap starting.
I20260808 02:26:04.115481 16152 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:42033
I20260808 02:26:04.115622 16328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:42033 every 8 connection(s)
I20260808 02:26:04.115949 16152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:04.116122 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16152
I20260808 02:26:04.116425 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:38365
--local_ip_for_outbound_sockets=127.30.160.2
--tserver_master_addrs=127.30.160.62:39831
--webserver_port=34679
--webserver_interface=127.30.160.2
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:26:04.119727 16211 log.cc:824] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:04.122355 16211 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:26:04.122699 16211 tablet_bootstrap.cc:492] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Bootstrap complete.
I20260808 02:26:04.124075 16211 ts_tablet_manager.cc:1397] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.016s
I20260808 02:26:04.125448 16211 raft_consensus.cc:348] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:04.125788 16211 raft_consensus.cc:729] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce410e069984f8ba0242f3d0fc2efe0, State: Initialized, Role: FOLLOWER
I20260808 02:26:04.125955 16211 consensus_queue.cc:260] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:04.126214 16211 ts_tablet_manager.cc:1428] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260808 02:26:04.170791 16332 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:04.118765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:37443)
0808 02:26:04.119166 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.119212 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:04.119671 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:04.170583 (+ 50912us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":101,"threads_started":1}
W20260808 02:26:04.171104 16329 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:39831 (0 consecutive failures): Network error: Failed to ping master at 127.30.160.62:39831: Client connection negotiation failed: client connection to 127.30.160.62:39831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:26:04.171485 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.171354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:56401)
0808 02:26:04.171410 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.171459 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:26:04.171819 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.171724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:54821)
0808 02:26:04.171772 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.171800 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 02:26:04.171965 16329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:26:04.202015 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:04.202235 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:04.202275 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:04.203423 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:04.203492 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:04.203538 16331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:04.205062 16331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:38365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=34679
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:04.205327 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:04.205564 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:04.210436 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:04.210443 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:04.210494 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:04.210633 16331 server_base.cc:1034] running on GCE node
I20260808 02:26:04.210858 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:04.211109 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:04.212244 16331 hybrid_clock.cc:648] HybridClock initialized: now 1786155964212223 us; error 37 us; skew 500 ppm
Aug 08 02:26:04 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:04.214711 16331 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:26:04.215157 16331 webserver.cc:460] Webserver started at http://127.30.160.2:34679/ using document root <none> and password file <none>
I20260808 02:26:04.215381 16331 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:04.215451 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:04.216818 16331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:04.217414 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.217578 16331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:04.217676 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "cacf84c4da3c4ed19f2db86fd8aed978"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "eb98630dc702a32fd7d9d1e23b029a68"
server_key_iv: "30f949b26d4121b9d4bc4fc7957e73f6"
server_key_version: "encryptionkey@0"
I20260808 02:26:04.218021 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:04.244334 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:04.246326 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:04.246714 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:04.247027 16331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:04.247366 16331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:04.247426 16331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.247467 16331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:04.247512 16331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.254263 16331 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38365
I20260808 02:26:04.254305 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38365 every 8 connection(s)
I20260808 02:26:04.254666 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:04.255849 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.255468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:53069)
0808 02:26:04.255631 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.255702 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260808 02:26:04.256248 16475 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:39831 (0 consecutive failures): Network error: Failed to ping master at 127.30.160.62:39831: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
I20260808 02:26:04.256520 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.256429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:48479)
0808 02:26:04.256476 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.256503 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 02:26:04.256820 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.256747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:47339)
0808 02:26:04.256784 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.256805 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 02:26:04.256937 16475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:26:04.263320 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16331
I20260808 02:26:04.263620 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:42073
--local_ip_for_outbound_sockets=127.30.160.3
--tserver_master_addrs=127.30.160.62:39831
--webserver_port=44031
--webserver_interface=127.30.160.3
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:04.346729 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:04.346933 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:04.346956 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:04.348191 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:04.348233 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:04.348284 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:04.349735 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=44031
--tserver_master_addrs=127.30.160.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:04.349985 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:04.350219 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:04.355201 16485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:04.355214 16484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:04.355214 16487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:04.355439 16478 server_base.cc:1034] running on GCE node
I20260808 02:26:04.355662 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:04.355997 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:04.357169 16478 hybrid_clock.cc:648] HybridClock initialized: now 1786155964357138 us; error 46 us; skew 500 ppm
Aug 08 02:26:04 dist-test-slave-zkpd krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:04.359696 16478 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:26:04.360164 16478 webserver.cc:460] Webserver started at http://127.30.160.3:44031/ using document root <none> and password file <none>
I20260808 02:26:04.360373 16478 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:04.360414 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:04.361649 16478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:04.362210 16494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.362427 16478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:04.362502 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "e1d6ce5e04a1485c8482a50a26d34aff"
format_stamp: "Formatted at 2026-08-08 02:25:22 on dist-test-slave-zkpd"
server_key: "b6c59d4b4deceb6cde242d80e95fd653"
server_key_iv: "d4934549a44f8d81b04b88ded67b5abd"
server_key_version: "encryptionkey@0"
I20260808 02:26:04.362810 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:04.380577 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:04.382382 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:04.382493 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:04.382660 16478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:04.382974 16478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:04.383008 16478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.383029 16478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:04.383077 16478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:04.389263 16478 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:42073
I20260808 02:26:04.389319 16614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:42073 every 8 connection(s)
I20260808 02:26:04.389680 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:26:04.390708 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.390356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:45207)
0808 02:26:04.390504 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.390569 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260808 02:26:04.391121 16615 heartbeater.cc:644] Failed to heartbeat to 127.30.160.62:39831 (0 consecutive failures): Network error: Failed to ping master at 127.30.160.62:39831: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
I20260808 02:26:04.391435 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.391343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:44523)
0808 02:26:04.391387 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.391416 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 02:26:04.391723 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:04.391663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:53525)
0808 02:26:04.391688 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:04.391709 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260808 02:26:04.391852 16615 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:26:04.399477 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16478
I20260808 02:26:05.172595 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.172363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:50519)
0808 02:26:05.172491 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:05.172528 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260808 02:26:05.257407 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.257194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:48263)
0808 02:26:05.257323 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:05.257357 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260808 02:26:05.392351 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.392144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:48585)
0808 02:26:05.392272 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:05.392305 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260808 02:26:05.854888 16623 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:05.855053 16623 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:05.855484 16623 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
Aug 08 02:26:05 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.2@KRBTEST.COM
Aug 08 02:26:05 dist-test-slave-zkpd krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.3@KRBTEST.COM
I20260808 02:26:05.866196 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.855862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:42073 (local address 127.30.160.1:55641)
0808 02:26:05.856013 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:05.856022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:05.856074 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:05.856835 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:05.856837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:05.857374 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:05.857572 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:05.857577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.858663 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.858664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:05.859449 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:05.859452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.860111 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.860112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:05.860210 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:05.860856 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:05.860861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:05.862728 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:05.864772 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:05.864773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:05.864773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:05.864949 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:05.865236 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:05.865237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:05.865237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:05.865301 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:05.865763 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:05.865763 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:05.865805 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:26:05.865881 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:26:05.866087 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":54,"thread_start_us":98,"threads_started":1}
I20260808 02:26:05.866215 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.855811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:36669 (local address 127.30.160.2:38365)
0808 02:26:05.856017 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:05.856032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:05.856065 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:26:05.856554 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:05.856556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:05.856602 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:05.856730 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:05.857712 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.858584 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.859541 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.860000 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.860122 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:05.862450 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:05.862454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:05.862454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:05.862464 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:05.864470 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:05.864899 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:05.864902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:05.864903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:05.864994 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:05.865254 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:05.865255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:05.865255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:05.865545 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:05.865605 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:05.865991 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 02:26:05.866112 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260808 02:26:05.866320 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.855665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38365 (local address 127.30.160.1:36669)
0808 02:26:05.855791 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:05.855802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:05.855861 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:05.856807 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:05.856810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:05.857375 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:05.857571 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:05.857577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.858701 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.858702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:05.859448 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:05.859450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.860100 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.860101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:05.860212 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:05.860805 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:05.860813 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:05.862340 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:05.864575 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:05.864582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:05.864585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:05.864798 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:05.865072 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:05.865073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:05.865073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:05.865182 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:05.865631 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:05.865638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:05.865787 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 02:26:05.865919 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 02:26:05.866171 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":139,"thread_start_us":82,"threads_started":1}
I20260808 02:26:05.866230 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:05.855988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:55641 (local address 127.30.160.3:42073)
0808 02:26:05.856141 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:05.856153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:05.856189 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:26:05.856618 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:05.856620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:05.856657 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:05.856817 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:05.857697 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.858562 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.859559 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:05.860006 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:05.860135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:05.862850 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:05.862853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:05.862853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:05.862863 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:05.864668 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:05.865041 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:05.865044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:05.865045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:05.865132 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:05.865405 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:05.865406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:05.865407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:05.865689 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:05.865769 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:05.865962 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 02:26:05.866110 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260808 02:26:05.867379 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
W20260808 02:26:05.867460 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:26:05.867513 16203 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce410e069984f8ba0242f3d0fc2efe0; no voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff
I20260808 02:26:05.867620 16623 raft_consensus.cc:2726] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:26:06.173204 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:06.173034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:51941)
0808 02:26:06.173119 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:06.173160 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:26:06.258138 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:06.257891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:45081)
0808 02:26:06.258047 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:06.258093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260808 02:26:06.393018 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:06.392824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:36881)
0808 02:26:06.392946 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:06.392980 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260808 02:26:07.174036 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:07.173781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:35549)
0808 02:26:07.173936 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:07.173986 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260808 02:26:07.258991 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:07.258758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:51249)
0808 02:26:07.258906 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:07.258952 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260808 02:26:07.388033 16632 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:07.388135 16632 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:07.388370 16632 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
W20260808 02:26:07.388868 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
W20260808 02:26:07.388983 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:26:07.389076 16203 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce410e069984f8ba0242f3d0fc2efe0; no voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff
I20260808 02:26:07.389189 16632 raft_consensus.cc:2726] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:26:07.393733 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:07.393519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:40405)
0808 02:26:07.393643 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:07.393683 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260808 02:26:08.174818 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:08.174582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:56183)
0808 02:26:08.174718 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:08.174755 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260808 02:26:08.259665 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:08.259428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:46333)
0808 02:26:08.259574 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:08.259609 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260808 02:26:08.394416 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:08.394190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:58575)
0808 02:26:08.394333 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:08.394369 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260808 02:26:09.175580 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:09.175332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:35379)
0808 02:26:09.175485 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:09.175526 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260808 02:26:09.260308 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:09.260112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:56279)
0808 02:26:09.260234 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:09.260270 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260808 02:26:09.346921 16639 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:09.347016 16639 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:09.347164 16639 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
W20260808 02:26:09.347596 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
W20260808 02:26:09.347683 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:26:09.347702 16203 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce410e069984f8ba0242f3d0fc2efe0; no voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff
I20260808 02:26:09.347780 16639 raft_consensus.cc:2726] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:26:09.395133 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:09.394900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:52789)
0808 02:26:09.395053 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:09.395090 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260808 02:26:10.176484 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:10.176238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:34605)
0808 02:26:10.176390 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:10.176431 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260808 02:26:10.260936 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:10.260725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:34155)
0808 02:26:10.260856 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:10.260894 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260808 02:26:10.395972 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:10.395730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:52581)
0808 02:26:10.395851 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:10.395922 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260808 02:26:11.177205 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:11.176986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:37647)
0808 02:26:11.177120 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:11.177156 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260808 02:26:11.261581 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:11.261364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:38631)
0808 02:26:11.261503 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:11.261541 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260808 02:26:11.396595 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:11.396390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:51097)
0808 02:26:11.396519 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:11.396554 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260808 02:26:12.177933 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:12.177632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:35677)
0808 02:26:12.177848 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:12.177885 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":60,"threads_started":1}
I20260808 02:26:12.262207 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:12.261999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:49687)
0808 02:26:12.262128 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:12.262165 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260808 02:26:12.397205 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:12.397009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:47309)
0808 02:26:12.397132 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:12.397165 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260808 02:26:13.178706 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:13.178499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:37895)
0808 02:26:13.178622 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:13.178657 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260808 02:26:13.262912 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:13.262700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:43595)
0808 02:26:13.262836 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:13.262872 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260808 02:26:13.397929 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:13.397729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:49759)
0808 02:26:13.397853 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:13.397888 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260808 02:26:13.957056 16654 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:13.957155 16654 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:13.957367 16654 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
W20260808 02:26:13.957746 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
W20260808 02:26:13.957823 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:26:13.957841 16203 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce410e069984f8ba0242f3d0fc2efe0; no voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff
I20260808 02:26:13.957923 16654 raft_consensus.cc:2726] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:26:14.179350 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:14.179142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:33703)
0808 02:26:14.179267 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:14.179301 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260808 02:26:14.263525 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:14.263318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:33271)
0808 02:26:14.263453 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:14.263488 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260808 02:26:14.398581 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:14.398368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:37471)
0808 02:26:14.398492 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:14.398522 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260808 02:26:15.180056 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:15.179787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:37087)
0808 02:26:15.179941 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:15.179981 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260808 02:26:15.264218 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:15.263983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:42833)
0808 02:26:15.264136 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:15.264171 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260808 02:26:15.399257 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:15.399044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:42505)
0808 02:26:15.399182 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:15.399214 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260808 02:26:16.180778 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:16.180561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:45067)
0808 02:26:16.180696 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:16.180729 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260808 02:26:16.264856 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:16.264647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:54347)
0808 02:26:16.264777 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:16.264820 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260808 02:26:16.399875 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:16.399681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:51451)
0808 02:26:16.399803 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:16.399834 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260808 02:26:17.181458 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:17.181240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:48225)
0808 02:26:17.181373 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:17.181409 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260808 02:26:17.265447 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:17.265241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:33633)
0808 02:26:17.265366 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:17.265402 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260808 02:26:17.400594 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:17.400340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:38355)
0808 02:26:17.400513 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:17.400548 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260808 02:26:17.628782 16667 raft_consensus.cc:482] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:17.628880 16667 raft_consensus.cc:504] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cacf84c4da3c4ed19f2db86fd8aed978" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38365 } } peers { permanent_uuid: "e1d6ce5e04a1485c8482a50a26d34aff" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42073 } } peers { permanent_uuid: "3ce410e069984f8ba0242f3d0fc2efe0" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 42033 } }
I20260808 02:26:17.629099 16667 leader_election.cc:288] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365), e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073)
W20260808 02:26:17.629540 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d6ce5e04a1485c8482a50a26d34aff (127.30.160.3:42073): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
W20260808 02:26:17.629662 16203 leader_election.cc:341] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cacf84c4da3c4ed19f2db86fd8aed978 (127.30.160.2:38365): Not found: Tablet not found: 01434be8a5d54d5088a1b76ffdd5ce3a
I20260808 02:26:17.629688 16203 leader_election.cc:302] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce410e069984f8ba0242f3d0fc2efe0; no voters: cacf84c4da3c4ed19f2db86fd8aed978, e1d6ce5e04a1485c8482a50a26d34aff
I20260808 02:26:17.629760 16667 raft_consensus.cc:2726] T 01434be8a5d54d5088a1b76ffdd5ce3a P 3ce410e069984f8ba0242f3d0fc2efe0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:26:18.182163 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:18.181935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:35573)
0808 02:26:18.182063 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:18.182100 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260808 02:26:18.266170 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:18.265896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:55541)
0808 02:26:18.266092 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:18.266129 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
I20260808 02:26:18.401269 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:18.401046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:52263)
0808 02:26:18.401176 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:18.401210 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260808 02:26:19.182935 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:19.182725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.1:53315)
0808 02:26:19.182855 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:19.182887 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260808 02:26:19.266884 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:19.266687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.2:34463)
0808 02:26:19.266811 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:19.266844 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260808 02:26:19.401998 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:19.401771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39831 (local address 127.30.160.3:56407)
0808 02:26:19.401914 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:19.401950 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.62:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260808 02:26:19.409374 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 16152
I20260808 02:26:19.416186 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 16331
I20260808 02:26:19.422288 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 16478
2026-08-08T02:26:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58379 ms)
[----------] 30 tests from SecurityITest (232836 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:19 dist-test-slave-zkpd krb5kdc[16676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:19 dist-test-slave-zkpd krb5kdc[16676](info): set up 2 sockets
Aug 08 02:26:19 dist-test-slave-zkpd krb5kdc[16676](info): commencing operation
krb5kdc: starting...
W20260808 02:26:20.508492 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:20 dist-test-slave-zkpd krb5kdc[16676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:20.630692 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39245
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:20.711977 16689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:20.712210 16689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:20.712245 16689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:20.713364 16689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:20.713438 16689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:20.713465 16689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:20.713500 16689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:20.713531 16689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:20.714982 16689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:20.715243 16689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:20.715467 16689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:20.720515 16694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:20.720554 16689 server_base.cc:1034] running on GCE node
W20260808 02:26:20.720561 16695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:20.720520 16697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:20.720979 16689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:20.721266 16689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:20.722446 16689 hybrid_clock.cc:648] HybridClock initialized: now 1786155980722425 us; error 41 us; skew 500 ppm
Aug 08 02:26:20 dist-test-slave-zkpd krb5kdc[16676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155980, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:20.725302 16689 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:20.725731 16689 webserver.cc:460] Webserver started at http://127.30.160.62:40331/ using document root <none> and password file <none>
I20260808 02:26:20.725956 16689 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:20.726027 16689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:20.726183 16689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:20.727257 16689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "7cdafdc656214e0db1ac574b84f67934"
format_stamp: "Formatted at 2026-08-08 02:26:20 on dist-test-slave-zkpd"
server_key: "522f4367c1c1099b9aa05ed0af57db27"
server_key_iv: "f652df617d0b46157519212376a544cf"
server_key_version: "encryptionkey@0"
I20260808 02:26:20.727721 16689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "7cdafdc656214e0db1ac574b84f67934"
format_stamp: "Formatted at 2026-08-08 02:26:20 on dist-test-slave-zkpd"
server_key: "522f4367c1c1099b9aa05ed0af57db27"
server_key_iv: "f652df617d0b46157519212376a544cf"
server_key_version: "encryptionkey@0"
I20260808 02:26:20.729136 16689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:26:20.729884 16704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:20.730060 16689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:20.730161 16689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "7cdafdc656214e0db1ac574b84f67934"
format_stamp: "Formatted at 2026-08-08 02:26:20 on dist-test-slave-zkpd"
server_key: "522f4367c1c1099b9aa05ed0af57db27"
server_key_iv: "f652df617d0b46157519212376a544cf"
server_key_version: "encryptionkey@0"
I20260808 02:26:20.730247 16689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:20.740375 16689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:20.742089 16689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:20.742245 16689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:20.746418 16689 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39245
I20260808 02:26:20.746457 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39245 every 8 connection(s)
I20260808 02:26:20.746815 16689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:20.747774 16765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:20.750327 16765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934: Bootstrap starting.
I20260808 02:26:20.750959 16765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:20.751277 16765 log.cc:824] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:20.752112 16765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934: No bootstrap required, opened a new log
I20260808 02:26:20.753861 16765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdafdc656214e0db1ac574b84f67934" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39245 } }
I20260808 02:26:20.754011 16765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:20.754073 16765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdafdc656214e0db1ac574b84f67934, State: Initialized, Role: FOLLOWER
I20260808 02:26:20.754235 16765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdafdc656214e0db1ac574b84f67934" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39245 } }
I20260808 02:26:20.754323 16765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:20.754381 16765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:20.754434 16765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:20.755043 16765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdafdc656214e0db1ac574b84f67934" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39245 } }
I20260808 02:26:20.755172 16765 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdafdc656214e0db1ac574b84f67934; no voters:
I20260808 02:26:20.755378 16765 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:20.755434 16768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:20.755586 16768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [term 1 LEADER]: Becoming Leader. State: Replica: 7cdafdc656214e0db1ac574b84f67934, State: Running, Role: LEADER
I20260808 02:26:20.755647 16765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:20.755712 16768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdafdc656214e0db1ac574b84f67934" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39245 } }
I20260808 02:26:20.755934 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16689
I20260808 02:26:20.756094 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:20.756356 31360 external_mini_cluster.cc:1428] Setting key 7805694debeb23b1b08a74fa857df10d
I20260808 02:26:20.756356 16770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdafdc656214e0db1ac574b84f67934. Latest consensus state: current_term: 1 leader_uuid: "7cdafdc656214e0db1ac574b84f67934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdafdc656214e0db1ac574b84f67934" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39245 } } }
I20260808 02:26:20.756433 16770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:20.756773 16777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:20.756785 16769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cdafdc656214e0db1ac574b84f67934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdafdc656214e0db1ac574b84f67934" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39245 } } }
I20260808 02:26:20.756971 16769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:20.757401 16777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:20.759146 16777 catalog_manager.cc:1347] Generated new cluster ID: 43c04500003c4a3fbfe9a2946d01139c
I20260808 02:26:20.759202 16777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:20 dist-test-slave-zkpd krb5kdc[16676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:20.768185 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:20.757036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.30.160.62:39245)
0808 02:26:20.757235 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:20.757246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:20.757476 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:26:20.758066 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:20.758073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:20.758163 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:20.759240 (+ 1077us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:20.759348 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:20.760428 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:20.761332 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:20.761804 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:20.761944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:20.764572 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:20.764576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:20.764577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:20.764589 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:20.766444 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:20.766874 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:20.766877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:20.766878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:20.766990 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:20.767242 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:20.767244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:20.767244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:20.767491 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:20.767681 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:20.767775 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:26:20.767986 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260808 02:26:20.775436 16777 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:20.776011 16777 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:20.780133 16777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934: Generated new TSK 0
I20260808 02:26:20.780292 16777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:26:20.810110 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39245
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:20.892889 16791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:20.893085 16791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:20.893108 16791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:20.894222 16791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:20.894280 16791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:20.894351 16791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:20.895753 16791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:20.896018 16791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:20.896204 16791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:20.901008 16800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:20.901001 16798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:20.900988 16797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:20.901034 16791 server_base.cc:1034] running on GCE node
I20260808 02:26:20.901338 16791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:20.901584 16791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:20.902745 16791 hybrid_clock.cc:648] HybridClock initialized: now 1786155980902727 us; error 46 us; skew 500 ppm
Aug 08 02:26:20 dist-test-slave-zkpd krb5kdc[16676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155980, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:20.905175 16791 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:26:20.905599 16791 webserver.cc:460] Webserver started at http://127.30.160.1:45217/ using document root <none> and password file <none>
I20260808 02:26:20.905840 16791 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:20.905912 16791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:20.906065 16791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:20.907110 16791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "6004c8b40aec40c0997e038b74119016"
format_stamp: "Formatted at 2026-08-08 02:26:20 on dist-test-slave-zkpd"
server_key: "ba27fe7f2ce5ae2d0709faa84f64f8b3"
server_key_iv: "276e45c5b5357acbe8b73c5ed6ce0454"
server_key_version: "encryptionkey@0"
I20260808 02:26:20.907480 16791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "6004c8b40aec40c0997e038b74119016"
format_stamp: "Formatted at 2026-08-08 02:26:20 on dist-test-slave-zkpd"
server_key: "ba27fe7f2ce5ae2d0709faa84f64f8b3"
server_key_iv: "276e45c5b5357acbe8b73c5ed6ce0454"
server_key_version: "encryptionkey@0"
I20260808 02:26:20.908854 16791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:20.909543 16808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:20.909703 16791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:20.909801 16791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "6004c8b40aec40c0997e038b74119016"
format_stamp: "Formatted at 2026-08-08 02:26:20 on dist-test-slave-zkpd"
server_key: "ba27fe7f2ce5ae2d0709faa84f64f8b3"
server_key_iv: "276e45c5b5357acbe8b73c5ed6ce0454"
server_key_version: "encryptionkey@0"
I20260808 02:26:20.909885 16791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:20.945900 16791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:20.947765 16791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:20.947976 16791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:20.948232 16791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:20.948552 16791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:20.948607 16791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:20.948647 16791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:20.948678 16791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:20.954269 16791 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:33967
I20260808 02:26:20.954357 16927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:33967 every 8 connection(s)
I20260808 02:26:20.954660 16791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:20.955132 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16791
I20260808 02:26:20.955274 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:20.955506 31360 external_mini_cluster.cc:1428] Setting key 900dd45506cf84072d23d082654ed299
Aug 08 02:26:20 dist-test-slave-zkpd krb5kdc[16676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155980, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
I20260808 02:26:20.964862 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:20.955373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:56115 (local address 127.30.160.62:39245)
0808 02:26:20.955430 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:20.955433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:20.955704 (+ 271us) server_negotiation.cc:407] Connection header received
0808 02:26:20.956418 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:20.956420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:20.956433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:20.956520 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:20.957459 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:20.958243 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:20.959654 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:20.959861 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:20.959951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:20.962039 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:20.962041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:20.962042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:20.962050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:20.963467 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:20.963867 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:20.963868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:20.963868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:20.963947 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:20.964212 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:20.964213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:20.964213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:20.964333 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:20.964409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:20.964683 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 02:26:20.964769 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:26:20.965075 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:20.955396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39245 (local address 127.30.160.1:56115)
0808 02:26:20.955563 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:20.955614 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:20.956320 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:20.956581 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:20.956584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:20.957188 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:20.957374 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:20.957378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:20.958338 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:20.958339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:20.959555 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:20.959557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:20.959968 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:20.959969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:20.960086 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:20.960597 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:20.960605 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:20.961944 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:20.963547 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:20.963554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:20.963557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:20.963780 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:20.964018 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:20.964019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:20.964019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:20.964134 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:20.964398 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:20.964405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:20.964550 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 02:26:20.964659 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:26:20.964820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260808 02:26:20.965530 16928 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39245
I20260808 02:26:20.965624 16928 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:20.965816 16928 heartbeater.cc:511] Master 127.30.160.62:39245 requested a full tablet report, sending...
I20260808 02:26:20.966300 16724 ts_manager.cc:194] Registered new tserver with Master: 6004c8b40aec40c0997e038b74119016 (127.30.160.1:33967)
I20260808 02:26:20.966925 16724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:56115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:26:20.980456 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39245
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:21.062582 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:21.062773 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:21.062803 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:21.063856 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:21.063907 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:21.063987 16933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:21.065480 16933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:21.065665 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:21.065847 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:21.070828 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.070802 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:21.070930 16933 server_base.cc:1034] running on GCE node
W20260808 02:26:21.070850 16939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:21.071219 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:21.071493 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:21.072649 16933 hybrid_clock.cc:648] HybridClock initialized: now 1786155981072631 us; error 28 us; skew 500 ppm
Aug 08 02:26:21 dist-test-slave-zkpd krb5kdc[16676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:21.075026 16933 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:26:21.075501 16933 webserver.cc:460] Webserver started at http://127.30.160.2:38167/ using document root <none> and password file <none>
I20260808 02:26:21.075737 16933 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:21.075806 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:21.075978 16933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:21.077003 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "1a9904d0e96342a3b43b1a104f74044c"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "acb3b9e5092afb93a855ec1e37cb60f3"
server_key_iv: "b03c7841785c3f5d5b2a32a184f53c67"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.077373 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "1a9904d0e96342a3b43b1a104f74044c"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "acb3b9e5092afb93a855ec1e37cb60f3"
server_key_iv: "b03c7841785c3f5d5b2a32a184f53c67"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.078660 16933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:21.079361 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.079536 16933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:21.079627 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "1a9904d0e96342a3b43b1a104f74044c"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "acb3b9e5092afb93a855ec1e37cb60f3"
server_key_iv: "b03c7841785c3f5d5b2a32a184f53c67"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.079712 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:21.095666 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:21.097477 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:21.097635 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:21.097867 16933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:21.098176 16933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:21.098232 16933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.098272 16933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:21.098390 16933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.104640 16933 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:37963
I20260808 02:26:21.104691 17065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:37963 every 8 connection(s)
I20260808 02:26:21.104977 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:21.105232 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 16933
I20260808 02:26:21.105371 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:21.105612 31360 external_mini_cluster.cc:1428] Setting key 869993cf2300d1b9827fc6341de14ad9
Aug 08 02:26:21 dist-test-slave-zkpd krb5kdc[16676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
I20260808 02:26:21.116030 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.105827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:59399 (local address 127.30.160.62:39245)
0808 02:26:21.105888 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.105890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.106129 (+ 239us) server_negotiation.cc:407] Connection header received
0808 02:26:21.106863 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.106864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.106874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.106929 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:21.108252 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.109031 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.110457 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.110643 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.110698 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.113127 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:21.113129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:21.113130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:21.113138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:21.114610 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:21.115010 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:21.115011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:21.115011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:21.115070 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:21.115322 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:21.115322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:21.115323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:21.115477 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:21.115548 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.115819 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.115939 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:26:21.116209 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.105809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39245 (local address 127.30.160.2:59399)
0808 02:26:21.105977 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.106023 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.106756 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.107000 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.107004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.107836 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:21.108151 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.108157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.109131 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.109132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.110357 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.110360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.110744 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.110745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.110914 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.111383 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:21.111394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:21.113031 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:21.114695 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:21.114703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:21.114705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:21.114931 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:21.115138 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:21.115139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:21.115140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:21.115247 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:21.115552 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:21.115559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:21.115692 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.115797 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.115981 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260808 02:26:21.116731 17066 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39245
I20260808 02:26:21.116832 17066 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:21.117004 17066 heartbeater.cc:511] Master 127.30.160.62:39245 requested a full tablet report, sending...
I20260808 02:26:21.117398 16724 ts_manager.cc:194] Registered new tserver with Master: 1a9904d0e96342a3b43b1a104f74044c (127.30.160.2:37963)
I20260808 02:26:21.117724 16724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:59399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:26:21.132506 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39245
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:21.214587 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:21.214833 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:21.214870 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:21.216043 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:21.216111 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:21.216154 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:21.217628 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:21.217897 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:21.218128 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:21.223153 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.223183 17078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.223183 17080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:21.223376 17071 server_base.cc:1034] running on GCE node
I20260808 02:26:21.223613 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:21.223937 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:21.225090 17071 hybrid_clock.cc:648] HybridClock initialized: now 1786155981225083 us; error 34 us; skew 500 ppm
Aug 08 02:26:21 dist-test-slave-zkpd krb5kdc[16676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:21.227634 17071 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:26:21.228124 17071 webserver.cc:460] Webserver started at http://127.30.160.3:38779/ using document root <none> and password file <none>
I20260808 02:26:21.228344 17071 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:21.228417 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:21.228564 17071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:21.229683 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "5879066d2bf1462c9a67a524c4a5e00b"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "dd857acb3ccedc8ada70724e36b4bb92"
server_key_iv: "4c2d97d3da35d38101bd70ddf2c52183"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.230062 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "5879066d2bf1462c9a67a524c4a5e00b"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "dd857acb3ccedc8ada70724e36b4bb92"
server_key_iv: "4c2d97d3da35d38101bd70ddf2c52183"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.231309 17071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:21.232123 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.232304 17071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:21.232399 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "5879066d2bf1462c9a67a524c4a5e00b"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "dd857acb3ccedc8ada70724e36b4bb92"
server_key_iv: "4c2d97d3da35d38101bd70ddf2c52183"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.232453 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:21.242652 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:21.244644 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:21.244763 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:21.244943 17071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:21.245201 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:21.245231 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.245252 17071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:21.245265 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.251122 17071 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:46009
I20260808 02:26:21.251195 17209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:46009 every 8 connection(s)
I20260808 02:26:21.251518 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:26:21.257345 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17071
I20260808 02:26:21.257496 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:21.257712 31360 external_mini_cluster.cc:1428] Setting key f7af50e116e4f6a0f05a58641c9e91b8
Aug 08 02:26:21 dist-test-slave-zkpd krb5kdc[16676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:21.260812 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.252312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:57225 (local address 127.30.160.62:39245)
0808 02:26:21.252395 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.252403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.252599 (+ 196us) server_negotiation.cc:407] Connection header received
0808 02:26:21.253129 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.253130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.253142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.253201 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:21.254163 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.254660 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.255684 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.255955 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.256017 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.258405 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:21.258407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:21.258407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:21.258416 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:21.259637 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:21.260046 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:21.260046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:21.260047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:21.260096 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:21.260284 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:21.260284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:21.260284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:21.260393 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:21.260440 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.260662 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.260750 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:26:21.260924 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.252272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:39245 (local address 127.30.160.3:57225)
0808 02:26:21.252459 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.252516 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.253022 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.253276 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.253280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.253864 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:21.254050 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.254054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.254764 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.254765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.255571 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.255573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.256028 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.256029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.256154 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.256577 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:21.256586 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:21.258324 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:21.259708 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:21.259714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:21.259716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:21.259993 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:21.260152 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:21.260153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:21.260153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:21.260245 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:21.260448 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:21.260452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:21.260560 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.260638 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.260751 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260808 02:26:21.261381 17210 heartbeater.cc:347] Connected to a master server at 127.30.160.62:39245
I20260808 02:26:21.261471 17210 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:21.261638 17210 heartbeater.cc:511] Master 127.30.160.62:39245 requested a full tablet report, sending...
I20260808 02:26:21.262001 16724 ts_manager.cc:194] Registered new tserver with Master: 5879066d2bf1462c9a67a524c4a5e00b (127.30.160.3:46009)
I20260808 02:26:21.262377 16724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:57225
I20260808 02:26:21.269438 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:21.276782 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.270685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.30.160.62:39245)
0808 02:26:21.270748 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.270750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.270833 (+ 83us) server_negotiation.cc:407] Connection header received
0808 02:26:21.270921 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.270922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.270931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.270981 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:21.271515 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.272053 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.272757 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.272966 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.273036 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.274212 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:21.274213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:21.274213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:21.274220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:21.275575 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:21.275980 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:21.275982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:21.275982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:21.276037 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:21.276244 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:21.276245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:21.276246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:21.276393 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:21.276495 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.276606 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.276682 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:26:21.278429 16724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:21.279974 16724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:21.285872 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.282147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:46009 (local address 127.0.0.1:59920)
0808 02:26:21.282533 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.282543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.282585 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.282990 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.282991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.282993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:21.283033 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.283034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.284012 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.284013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.284914 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.284917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.285582 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.285583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.285666 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.285679 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.285747 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.285799 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260808 02:26:21.285881 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.282523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.30.160.3:46009)
0808 02:26:21.282790 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.282794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.282803 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:21.282839 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.282841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.282856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.282942 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:21.283117 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.283850 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.285006 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.285495 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.285580 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.285640 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.285765 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.285806 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260808 02:26:21.286716 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.282277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.30.160.2:37963)
0808 02:26:21.282531 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.282536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.282548 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:26:21.282593 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.282595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.282611 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.282700 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:21.282878 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.283736 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.285627 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.286353 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.286426 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.286473 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.286586 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.286635 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260808 02:26:21.286907 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.282147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:37963 (local address 127.0.0.1:44168)
0808 02:26:21.282262 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.282328 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.282399 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.282712 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.282716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.282719 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:21.282790 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.282791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.283847 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.283848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.285541 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.285543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.286442 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.286444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.286505 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.286510 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.286569 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.286751 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":60,"thread_start_us":54,"threads_started":1}
I20260808 02:26:21.286782 17140 tablet_service.cc:1467] Processing CreateTablet for tablet ad5777b051df4cbaafd844d950f91939 (DEFAULT_TABLE table=test-table [id=a5d001c9926b4f2cac3dabcd05109640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:21.287197 17140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5777b051df4cbaafd844d950f91939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:21.287774 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.282342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:33967 (local address 127.0.0.1:60504)
0808 02:26:21.282702 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.282713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.282758 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.284149 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.284150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.284152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:21.284216 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.284218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.285494 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.285495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.286585 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.286588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.287539 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.287540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.287601 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.287605 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.287663 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.287703 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260808 02:26:21.287922 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.282573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.30.160.1:33967)
0808 02:26:21.283975 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.283979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.283989 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:21.284037 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.284043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.284058 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.284138 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:21.284299 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.285388 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.286684 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.287436 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.287535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.287590 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.287705 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.287748 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":96,"threads_started":1}
I20260808 02:26:21.288085 16999 tablet_service.cc:1467] Processing CreateTablet for tablet ad5777b051df4cbaafd844d950f91939 (DEFAULT_TABLE table=test-table [id=a5d001c9926b4f2cac3dabcd05109640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:21.288410 16999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5777b051df4cbaafd844d950f91939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:21.288664 16858 tablet_service.cc:1467] Processing CreateTablet for tablet ad5777b051df4cbaafd844d950f91939 (DEFAULT_TABLE table=test-table [id=a5d001c9926b4f2cac3dabcd05109640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:21.288977 16858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5777b051df4cbaafd844d950f91939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:21.290692 17227 tablet_bootstrap.cc:492] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Bootstrap starting.
I20260808 02:26:21.291288 17228 tablet_bootstrap.cc:492] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Bootstrap starting.
I20260808 02:26:21.291471 17227 tablet_bootstrap.cc:654] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:21.292248 17228 tablet_bootstrap.cc:654] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:21.292402 17229 tablet_bootstrap.cc:492] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: Bootstrap starting.
I20260808 02:26:21.292775 17227 log.cc:824] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:21.292893 17228 log.cc:824] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:21.293036 17229 tablet_bootstrap.cc:654] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:21.293617 17227 tablet_bootstrap.cc:492] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: No bootstrap required, opened a new log
I20260808 02:26:21.293645 17229 log.cc:824] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:21.293718 17227 ts_tablet_manager.cc:1397] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:26:21.293725 17228 tablet_bootstrap.cc:492] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: No bootstrap required, opened a new log
I20260808 02:26:21.293785 17228 ts_tablet_manager.cc:1397] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:26:21.294435 17229 tablet_bootstrap.cc:492] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: No bootstrap required, opened a new log
I20260808 02:26:21.294503 17229 ts_tablet_manager.cc:1397] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:21.295265 17227 raft_consensus.cc:348] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.295429 17227 raft_consensus.cc:374] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:21.295621 17227 raft_consensus.cc:729] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5879066d2bf1462c9a67a524c4a5e00b, State: Initialized, Role: FOLLOWER
I20260808 02:26:21.295819 17227 consensus_queue.cc:260] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.295756 17228 raft_consensus.cc:348] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.295962 17228 raft_consensus.cc:374] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:21.296011 17228 raft_consensus.cc:729] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9904d0e96342a3b43b1a104f74044c, State: Initialized, Role: FOLLOWER
I20260808 02:26:21.296111 17228 consensus_queue.cc:260] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.296165 17227 ts_tablet_manager.cc:1428] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:21.296288 17210 heartbeater.cc:503] Master 127.30.160.62:39245 was elected leader, sending a full tablet report...
I20260808 02:26:21.296363 17228 ts_tablet_manager.cc:1428] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:26:21.296481 17066 heartbeater.cc:503] Master 127.30.160.62:39245 was elected leader, sending a full tablet report...
I20260808 02:26:21.296766 17229 raft_consensus.cc:348] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.296917 17229 raft_consensus.cc:374] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:21.296940 17229 raft_consensus.cc:729] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6004c8b40aec40c0997e038b74119016, State: Initialized, Role: FOLLOWER
I20260808 02:26:21.297049 17229 consensus_queue.cc:260] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.297323 17229 ts_tablet_manager.cc:1428] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:26:21.297350 16928 heartbeater.cc:503] Master 127.30.160.62:39245 was elected leader, sending a full tablet report...
W20260808 02:26:21.355568 17067 tablet.cc:2369] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:26:21.456172 16929 tablet.cc:2369] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:26:21.502185 17211 tablet.cc:2369] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:21.550971 17234 raft_consensus.cc:482] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:21.551102 17234 raft_consensus.cc:504] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.551568 17234 leader_election.cc:288] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5879066d2bf1462c9a67a524c4a5e00b (127.30.160.3:46009), 6004c8b40aec40c0997e038b74119016 (127.30.160.1:33967)
I20260808 02:26:21.554741 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.551995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:34211 (local address 127.30.160.1:33967)
0808 02:26:21.552045 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.552047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.552291 (+ 244us) server_negotiation.cc:407] Connection header received
0808 02:26:21.552324 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.552326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.552339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.552399 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:21.552594 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.553087 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.554008 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.554377 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.554434 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.554465 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.554615 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.554664 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:26:21.554744 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.551876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:33967 (local address 127.30.160.2:34211)
0808 02:26:21.552143 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.552154 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.552217 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.552443 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.552443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.552445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:21.552518 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.552519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.553169 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.553170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.553912 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.553914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.554466 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.554467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.554545 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.554548 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.554590 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.554618 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":66,"thread_start_us":138,"threads_started":1}
I20260808 02:26:21.554800 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.552012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:38761 (local address 127.30.160.3:46009)
0808 02:26:21.552055 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.552058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.552066 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:21.552099 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.552100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.552111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.552198 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:21.552443 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.553002 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.554006 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.554382 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.554478 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.554485 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.554671 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.554718 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:26:21.554827 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.551876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:46009 (local address 127.30.160.2:38761)
0808 02:26:21.551973 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.551986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.552038 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.552270 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.552272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.552274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:21.552362 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.552364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.553087 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.553089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.553906 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.553908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.554492 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.554494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.554570 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.554590 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.554649 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.554686 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":3}
I20260808 02:26:21.555126 16879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5777b051df4cbaafd844d950f91939" candidate_uuid: "1a9904d0e96342a3b43b1a104f74044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6004c8b40aec40c0997e038b74119016" is_pre_election: true
I20260808 02:26:21.555198 17160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5777b051df4cbaafd844d950f91939" candidate_uuid: "1a9904d0e96342a3b43b1a104f74044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5879066d2bf1462c9a67a524c4a5e00b" is_pre_election: true
I20260808 02:26:21.555311 16879 raft_consensus.cc:2454] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9904d0e96342a3b43b1a104f74044c in term 0.
I20260808 02:26:21.555388 17160 raft_consensus.cc:2454] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9904d0e96342a3b43b1a104f74044c in term 0.
I20260808 02:26:21.555577 16955 leader_election.cc:302] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9904d0e96342a3b43b1a104f74044c, 6004c8b40aec40c0997e038b74119016; no voters:
I20260808 02:26:21.555761 17234 raft_consensus.cc:2781] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:21.555838 17234 raft_consensus.cc:482] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:21.555873 17234 raft_consensus.cc:3037] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:21.556767 17234 raft_consensus.cc:504] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.556932 17234 leader_election.cc:288] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [CANDIDATE]: Term 1 election: Requested vote from peers 5879066d2bf1462c9a67a524c4a5e00b (127.30.160.3:46009), 6004c8b40aec40c0997e038b74119016 (127.30.160.1:33967)
I20260808 02:26:21.557142 16879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5777b051df4cbaafd844d950f91939" candidate_uuid: "1a9904d0e96342a3b43b1a104f74044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6004c8b40aec40c0997e038b74119016"
I20260808 02:26:21.557142 17160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5777b051df4cbaafd844d950f91939" candidate_uuid: "1a9904d0e96342a3b43b1a104f74044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5879066d2bf1462c9a67a524c4a5e00b"
I20260808 02:26:21.557224 17160 raft_consensus.cc:3037] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:21.557224 16879 raft_consensus.cc:3037] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:21.558126 17160 raft_consensus.cc:2454] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9904d0e96342a3b43b1a104f74044c in term 1.
I20260808 02:26:21.558126 16879 raft_consensus.cc:2454] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9904d0e96342a3b43b1a104f74044c in term 1.
I20260808 02:26:21.558333 16955 leader_election.cc:302] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9904d0e96342a3b43b1a104f74044c, 6004c8b40aec40c0997e038b74119016; no voters:
I20260808 02:26:21.558446 17234 raft_consensus.cc:2781] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:21.558652 17234 raft_consensus.cc:686] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 1 LEADER]: Becoming Leader. State: Replica: 1a9904d0e96342a3b43b1a104f74044c, State: Running, Role: LEADER
I20260808 02:26:21.558764 17234 consensus_queue.cc:237] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } }
I20260808 02:26:21.559630 16724 catalog_manager.cc:5495] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9904d0e96342a3b43b1a104f74044c (127.30.160.2). New cstate: current_term: 1 leader_uuid: "1a9904d0e96342a3b43b1a104f74044c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9904d0e96342a3b43b1a104f74044c" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 } health_report { overall_health: UNKNOWN } } }
I20260808 02:26:21.570272 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.567859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.30.160.2:37963)
0808 02:26:21.567920 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.567922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.567931 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:21.567966 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.567967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.567986 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.568074 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:21.568206 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.568670 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.569435 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.569639 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.569730 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.569900 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:21.570043 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:21.570132 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.570158 (+ 26us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.570177 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:26:21.572083 17160 raft_consensus.cc:1261] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 1 FOLLOWER]: Refusing update from remote peer 1a9904d0e96342a3b43b1a104f74044c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:21.572083 16879 raft_consensus.cc:1261] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016 [term 1 FOLLOWER]: Refusing update from remote peer 1a9904d0e96342a3b43b1a104f74044c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:21.572431 17234 consensus_queue.cc:1035] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5879066d2bf1462c9a67a524c4a5e00b" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:21.572561 17237 consensus_queue.cc:1035] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6004c8b40aec40c0997e038b74119016" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 33967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:21.574993 17240 mvcc.cc:205] Tried to move back new op lower bound from 7316094900516134912 to 7316094900465209344. Current Snapshot: MvccSnapshot[applied={T|T < 7316094900516134912}]
I20260808 02:26:21.577548 16723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:21.577626 16723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:21.578368 16723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cdafdc656214e0db1ac574b84f67934: Sending DeleteTablet for 3 replicas of tablet ad5777b051df4cbaafd844d950f91939
I20260808 02:26:21.578691 17140 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5777b051df4cbaafd844d950f91939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:21 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:59920
I20260808 02:26:21.578708 16858 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5777b051df4cbaafd844d950f91939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:21 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:60504
I20260808 02:26:21.578899 16999 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5777b051df4cbaafd844d950f91939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:21 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:44168
I20260808 02:26:21.578946 17250 tablet_replica.cc:330] T ad5777b051df4cbaafd844d950f91939 P 6004c8b40aec40c0997e038b74119016: stopping tablet replica
I20260808 02:26:21.578982 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 16791
I20260808 02:26:21.580454 17251 tablet_replica.cc:330] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: stopping tablet replica
I20260808 02:26:21.580570 17251 raft_consensus.cc:2229] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:21.580683 17251 raft_consensus.cc:2258] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:21.581921 17251 ts_tablet_manager.cc:1905] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:21.583542 17251 ts_tablet_manager.cc:1918] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:21.583595 17251 log.cc:1196] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/ad5777b051df4cbaafd844d950f91939
I20260808 02:26:21.583810 17251 ts_tablet_manager.cc:1939] T ad5777b051df4cbaafd844d950f91939 P 5879066d2bf1462c9a67a524c4a5e00b: Deleting consensus metadata
I20260808 02:26:21.584195 16707 catalog_manager.cc:4841] TS 5879066d2bf1462c9a67a524c4a5e00b (127.30.160.3:46009): tablet ad5777b051df4cbaafd844d950f91939 (table test-table [id=a5d001c9926b4f2cac3dabcd05109640]) successfully deleted
I20260808 02:26:21.588297 17252 tablet_replica.cc:330] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: stopping tablet replica
I20260808 02:26:21.588444 17252 raft_consensus.cc:2229] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 1 LEADER]: Raft consensus shutting down.
W20260808 02:26:21.588423 16708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:33967 (error 108)
I20260808 02:26:21.588555 17252 raft_consensus.cc:2258] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:21.589026 17252 ts_tablet_manager.cc:1905] T ad5777b051df4cbaafd844d950f91939 P 1a9904d0e96342a3b43b1a104f74044c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:21.589071 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.588924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:33967 (local address 127.0.0.1:60520)
0808 02:26:21.588988 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.589033 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:33967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:26:21.589188 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 16933
W20260808 02:26:21.592545 16708 catalog_manager.cc:4584] TS 6004c8b40aec40c0997e038b74119016 (127.30.160.1:33967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad5777b051df4cbaafd844d950f91939: Network error: Client connection negotiation failed: client connection to 127.30.160.1:33967: connect: Connection refused (error 111)
I20260808 02:26:21.596201 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.596074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:37963 (local address 127.0.0.1:44176)
0808 02:26:21.596129 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.596163 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.2:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:26:21.596197 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17071
I20260808 02:26:21.601382 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 16689
2026-08-08T02:26:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2179 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T02:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:21Z Disabled control of system clock
I20260808 02:26:21.630877 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34987
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:21.712080 17257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:21.712285 17257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:21.712306 17257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:21.713367 17257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:21.713402 17257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:21.713413 17257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:21.713425 17257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:21.713438 17257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:21.714741 17257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:34987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:21.714927 17257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:21.715147 17257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:21.719832 17264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.719846 17263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.719869 17266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:21.720139 17257 server_base.cc:1034] running on GCE node
I20260808 02:26:21.720369 17257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:21.720642 17257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:21.721804 17257 hybrid_clock.cc:648] HybridClock initialized: now 1786155981721792 us; error 29 us; skew 500 ppm
I20260808 02:26:21.723177 17257 webserver.cc:460] Webserver started at http://127.30.160.62:41209/ using document root <none> and password file <none>
I20260808 02:26:21.723403 17257 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:21.723474 17257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:21.723634 17257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:21.724702 17257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "76a2b739d56f49d2ae37b3d69df44421"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "ecaadbc22c407fc9603ffa84e2d413a8"
server_key_iv: "0cb0f83298ddb7aef72936f77db9b937"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.725071 17257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "76a2b739d56f49d2ae37b3d69df44421"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "ecaadbc22c407fc9603ffa84e2d413a8"
server_key_iv: "0cb0f83298ddb7aef72936f77db9b937"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.726333 17257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:26:21.727013 17272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.727180 17257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:21.727277 17257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "76a2b739d56f49d2ae37b3d69df44421"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "ecaadbc22c407fc9603ffa84e2d413a8"
server_key_iv: "0cb0f83298ddb7aef72936f77db9b937"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.727358 17257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:21.746950 17257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:21.747254 17257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:21.747403 17257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:21.751108 17257 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:34987
I20260808 02:26:21.751183 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:34987 every 8 connection(s)
I20260808 02:26:21.751510 17257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:21.752490 17328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:21.754726 17328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421: Bootstrap starting.
I20260808 02:26:21.755275 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17257
I20260808 02:26:21.755404 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:21.755422 17328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:21.755631 31360 external_mini_cluster.cc:1428] Setting key c680f1e8066a55e34a15d0aec8fe3982
I20260808 02:26:21.755759 17328 log.cc:824] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:21.756524 17328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421: No bootstrap required, opened a new log
I20260808 02:26:21.758445 17328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2b739d56f49d2ae37b3d69df44421" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34987 } }
I20260808 02:26:21.758574 17328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:21.758595 17328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a2b739d56f49d2ae37b3d69df44421, State: Initialized, Role: FOLLOWER
I20260808 02:26:21.758713 17328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2b739d56f49d2ae37b3d69df44421" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34987 } }
I20260808 02:26:21.758788 17328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:21.758812 17328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:21.758876 17328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:21.759404 17328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2b739d56f49d2ae37b3d69df44421" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34987 } }
I20260808 02:26:21.759522 17328 leader_election.cc:302] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a2b739d56f49d2ae37b3d69df44421; no voters:
I20260808 02:26:21.759698 17328 leader_election.cc:288] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:21.759859 17334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:21.759999 17328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:21.760327 17334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [term 1 LEADER]: Becoming Leader. State: Replica: 76a2b739d56f49d2ae37b3d69df44421, State: Running, Role: LEADER
I20260808 02:26:21.760428 17334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2b739d56f49d2ae37b3d69df44421" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34987 } }
I20260808 02:26:21.760780 17333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a2b739d56f49d2ae37b3d69df44421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2b739d56f49d2ae37b3d69df44421" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34987 } } }
I20260808 02:26:21.760845 17333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:21.760995 17335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a2b739d56f49d2ae37b3d69df44421. Latest consensus state: current_term: 1 leader_uuid: "76a2b739d56f49d2ae37b3d69df44421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2b739d56f49d2ae37b3d69df44421" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34987 } } }
I20260808 02:26:21.761071 17335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:21.761430 17342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:21.761495 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.756133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.30.160.62:34987)
0808 02:26:21.756330 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.756340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.756371 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:26:21.756833 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.756840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.756897 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.757039 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:21.757250 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.758103 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.759110 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.759528 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.759643 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.760494 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:21.760499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:21.760499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:21.760723 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:21.760768 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:21.760795 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:21.760849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.761182 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.761275 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260808 02:26:21.761991 17342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:21.763566 17342 catalog_manager.cc:1347] Generated new cluster ID: 822d2d9981c9446eaae69466d31307a9
I20260808 02:26:21.763631 17342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:21.770933 17342 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:21.771485 17342 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:21.775676 17342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421: Generated new TSK 0
I20260808 02:26:21.775838 17342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:21.780901 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34987
--builtin_ntp_servers=127.30.160.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:21.864813 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:21.865012 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:21.865032 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:21.866225 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:21.866264 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:21.866289 17353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:21.867600 17353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:21.867835 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:21.868186 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:21.873930 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.873952 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.873955 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:21.874450 17353 server_base.cc:1034] running on GCE node
I20260808 02:26:21.874665 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:21.874917 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:21.876077 17353 hybrid_clock.cc:648] HybridClock initialized: now 1786155981876061 us; error 33 us; skew 500 ppm
I20260808 02:26:21.877368 17353 webserver.cc:460] Webserver started at http://127.30.160.1:43445/ using document root <none> and password file <none>
I20260808 02:26:21.877647 17353 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:21.877753 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:21.877905 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:21.878974 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "57aec456d81e418d97a41c970c38c6a9"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "52aa97e96d26368f06ab8d978d6f2c1c"
server_key_iv: "b8d4481c9f91f252283687cdd43f3a00"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.879345 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "57aec456d81e418d97a41c970c38c6a9"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "52aa97e96d26368f06ab8d978d6f2c1c"
server_key_iv: "b8d4481c9f91f252283687cdd43f3a00"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.880821 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:21.881582 17367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.881781 17353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:26:21.881882 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "57aec456d81e418d97a41c970c38c6a9"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "52aa97e96d26368f06ab8d978d6f2c1c"
server_key_iv: "b8d4481c9f91f252283687cdd43f3a00"
server_key_version: "encryptionkey@0"
I20260808 02:26:21.881965 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:21.898104 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:21.898419 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:21.898578 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:21.898810 17353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:21.899132 17353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:21.899186 17353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.899226 17353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:21.899256 17353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:21.905087 17353 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:37925
I20260808 02:26:21.905135 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:37925 every 8 connection(s)
I20260808 02:26:21.905452 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:21.906605 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17353
I20260808 02:26:21.906716 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:21.906898 31360 external_mini_cluster.cc:1428] Setting key 7880bdc3470c1ca52c81a7bda7450636
I20260808 02:26:21.907855 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34987
--builtin_ntp_servers=127.30.160.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:26:21.911612 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.906181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:41533 (local address 127.30.160.62:34987)
0808 02:26:21.906249 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:21.906251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:21.906562 (+ 311us) server_negotiation.cc:407] Connection header received
0808 02:26:21.907185 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:21.907187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:21.907200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:21.907275 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:21.908193 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.908928 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.910244 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.910435 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.910491 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.911341 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:21.911342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:21.911342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:21.911346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:21.911358 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:21.911362 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:21.911427 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:21.911534 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 02:26:21.911554 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:26:21.911950 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:21.906164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34987 (local address 127.30.160.1:41533)
0808 02:26:21.906399 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:21.906438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:21.907044 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:21.907383 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:21.907388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:21.907398 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:21.908094 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.908101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.909023 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.909023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.910150 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:21.910153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:21.910528 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:21.910530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:21.910681 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:21.911190 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:21.911190 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:21.911227 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:21.911255 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:21.911435 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:21.911440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:21.911443 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:21.911525 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:26:21.911634 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260808 02:26:21.912446 17485 heartbeater.cc:347] Connected to a master server at 127.30.160.62:34987
I20260808 02:26:21.912572 17485 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:21.912779 17485 heartbeater.cc:511] Master 127.30.160.62:34987 requested a full tablet report, sending...
I20260808 02:26:21.913311 17290 ts_manager.cc:194] Registered new tserver with Master: 57aec456d81e418d97a41c970c38c6a9 (127.30.160.1:37925)
I20260808 02:26:21.913924 17290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.1:41533
W20260808 02:26:21.991539 17488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:21.991721 17488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:21.991741 17488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:21.992915 17488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:21.992956 17488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:21.992980 17488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:21.994356 17488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:21.994590 17488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:21.994885 17488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:21.999845 17493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.999850 17495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:21.999855 17497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:22.000116 17488 server_base.cc:1034] running on GCE node
I20260808 02:26:22.000329 17488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:22.000576 17488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:22.001716 17488 hybrid_clock.cc:648] HybridClock initialized: now 1786155982001698 us; error 33 us; skew 500 ppm
I20260808 02:26:22.003177 17488 webserver.cc:460] Webserver started at http://127.30.160.2:34625/ using document root <none> and password file <none>
I20260808 02:26:22.003429 17488 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:22.003500 17488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:22.003639 17488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:22.004752 17488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "6182176ba8ad4d5c89fa40ee1e825da8"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "889dc11102d33ae80be3be23bf748b39"
server_key_iv: "b8de17f78ea1fa7d1be9cc6c821a38c0"
server_key_version: "encryptionkey@0"
I20260808 02:26:22.005119 17488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "6182176ba8ad4d5c89fa40ee1e825da8"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "889dc11102d33ae80be3be23bf748b39"
server_key_iv: "b8de17f78ea1fa7d1be9cc6c821a38c0"
server_key_version: "encryptionkey@0"
I20260808 02:26:22.006402 17488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:26:22.007124 17503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:22.007303 17488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:22.007400 17488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "6182176ba8ad4d5c89fa40ee1e825da8"
format_stamp: "Formatted at 2026-08-08 02:26:21 on dist-test-slave-zkpd"
server_key: "889dc11102d33ae80be3be23bf748b39"
server_key_iv: "b8de17f78ea1fa7d1be9cc6c821a38c0"
server_key_version: "encryptionkey@0"
I20260808 02:26:22.007488 17488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:22.036882 17488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:22.037209 17488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:22.037369 17488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:22.037612 17488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:22.037966 17488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:22.038028 17488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:22.038064 17488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:22.038091 17488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:22.043727 17488 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:43395
I20260808 02:26:22.043774 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:43395 every 8 connection(s)
I20260808 02:26:22.044154 17488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:22.044387 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17488
I20260808 02:26:22.044525 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:22.044751 31360 external_mini_cluster.cc:1428] Setting key a2b7eb3b28f910c221c99409955ea113
I20260808 02:26:22.045792 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34987
--builtin_ntp_servers=127.30.160.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:26:22.050107 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.044896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:40431 (local address 127.30.160.62:34987)
0808 02:26:22.044955 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.044958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.045271 (+ 313us) server_negotiation.cc:407] Connection header received
0808 02:26:22.045883 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.045885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.045898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.045965 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:22.046908 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.047384 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.048509 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.048734 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.048792 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.049763 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:22.049765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:22.049765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:22.049775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:22.049792 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:22.049799 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:22.049876 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.049988 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.050019 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:26:22.050362 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.044903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34987 (local address 127.30.160.2:40431)
0808 02:26:22.045131 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.045197 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.045775 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.046098 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.046103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.046113 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:22.046813 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.046820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.047510 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.047511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.048426 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.048429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.048835 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.048836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.048948 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.049556 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:22.049557 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:22.049589 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:22.049613 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:22.049881 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:22.049888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:22.049892 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.049990 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.050105 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260808 02:26:22.050930 17623 heartbeater.cc:347] Connected to a master server at 127.30.160.62:34987
I20260808 02:26:22.051038 17623 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:22.051252 17623 heartbeater.cc:511] Master 127.30.160.62:34987 requested a full tablet report, sending...
I20260808 02:26:22.051641 17290 ts_manager.cc:194] Registered new tserver with Master: 6182176ba8ad4d5c89fa40ee1e825da8 (127.30.160.2:43395)
I20260808 02:26:22.052150 17290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.2:40431
W20260808 02:26:22.128955 17626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:22.129141 17626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:22.129163 17626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:22.130345 17626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:22.130384 17626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:22.130409 17626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:22.131879 17626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:22.132095 17626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:22.132328 17626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:22.137092 17638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:22.137081 17634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:22.137110 17626 server_base.cc:1034] running on GCE node
W20260808 02:26:22.137094 17635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:22.137490 17626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:22.137751 17626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:22.138895 17626 hybrid_clock.cc:648] HybridClock initialized: now 1786155982138881 us; error 28 us; skew 500 ppm
I20260808 02:26:22.140079 17626 webserver.cc:460] Webserver started at http://127.30.160.3:46677/ using document root <none> and password file <none>
I20260808 02:26:22.140321 17626 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:22.140367 17626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:22.140524 17626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:22.141577 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "0ae077be39034ff7b8acef1cdba607e7"
format_stamp: "Formatted at 2026-08-08 02:26:22 on dist-test-slave-zkpd"
server_key: "1bec44881b3972e8510d53cf4c322675"
server_key_iv: "670e3cc0e801bbeb4dc0b5c041c116d8"
server_key_version: "encryptionkey@0"
I20260808 02:26:22.141920 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "0ae077be39034ff7b8acef1cdba607e7"
format_stamp: "Formatted at 2026-08-08 02:26:22 on dist-test-slave-zkpd"
server_key: "1bec44881b3972e8510d53cf4c322675"
server_key_iv: "670e3cc0e801bbeb4dc0b5c041c116d8"
server_key_version: "encryptionkey@0"
I20260808 02:26:22.143164 17626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:22.143850 17644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:22.144026 17626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:22.144124 17626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "0ae077be39034ff7b8acef1cdba607e7"
format_stamp: "Formatted at 2026-08-08 02:26:22 on dist-test-slave-zkpd"
server_key: "1bec44881b3972e8510d53cf4c322675"
server_key_iv: "670e3cc0e801bbeb4dc0b5c041c116d8"
server_key_version: "encryptionkey@0"
I20260808 02:26:22.144201 17626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:22.181504 17626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:22.181854 17626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:22.182015 17626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:22.182260 17626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:22.182624 17626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:22.182680 17626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:22.182726 17626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:22.182751 17626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:22.188678 17626 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:37501
I20260808 02:26:22.188727 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:37501 every 8 connection(s)
I20260808 02:26:22.189046 17626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:26:22.192369 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17626
I20260808 02:26:22.192472 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:22.192653 31360 external_mini_cluster.cc:1428] Setting key 31c66ea2311358c27b2779e566180c5f
I20260808 02:26:22.194542 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.189771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:53179 (local address 127.30.160.62:34987)
0808 02:26:22.189818 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.189820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.190060 (+ 240us) server_negotiation.cc:407] Connection header received
0808 02:26:22.190529 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.190530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.190540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.190588 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:22.191253 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.191750 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.193041 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.193316 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.193400 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.194261 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:22.194263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:22.194263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:22.194268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:22.194283 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:22.194287 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:22.194334 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.194440 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.194466 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:26:22.194762 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.189764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34987 (local address 127.30.160.3:53179)
0808 02:26:22.189945 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.189977 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.190449 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.190648 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.190651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.190659 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:22.191152 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.191156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.191835 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.191836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.192938 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.192941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.193424 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.193425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.193551 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.194108 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:22.194108 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:22.194137 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:22.194158 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:22.194334 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:22.194341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:22.194344 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.194440 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.194536 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260808 02:26:22.195286 17761 heartbeater.cc:347] Connected to a master server at 127.30.160.62:34987
I20260808 02:26:22.195358 17761 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:22.195547 17761 heartbeater.cc:511] Master 127.30.160.62:34987 requested a full tablet report, sending...
I20260808 02:26:22.196010 17290 ts_manager.cc:194] Registered new tserver with Master: 0ae077be39034ff7b8acef1cdba607e7 (127.30.160.3:37501)
I20260808 02:26:22.196483 17290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.3:53179
I20260808 02:26:22.204425 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:22.208432 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.205823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.30.160.62:34987)
0808 02:26:22.205865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.205867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.205977 (+ 110us) server_negotiation.cc:407] Connection header received
0808 02:26:22.206029 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.206029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.206037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.206072 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:22.206207 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.206648 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.207386 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.207561 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.207609 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.208244 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:22.208245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:22.208245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:22.208249 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:22.208260 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:22.208263 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:22.208307 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.208356 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.208373 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 02:26:22.209841 17290 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:22.211544 17290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:22.218026 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.213813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.30.160.3:37501)
0808 02:26:22.213964 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.213969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.213981 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:26:22.214020 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.214022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.214038 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.214128 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:22.214353 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.215503 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217093 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.217575 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217676 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.217766 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.217885 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.217933 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260808 02:26:22.218154 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.214084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.30.160.2:43395)
0808 02:26:22.214273 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.214277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.214287 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:22.214327 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.214329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.214344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.214424 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:22.214564 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.215639 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217070 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.217759 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217822 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.217900 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.217969 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.218007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
I20260808 02:26:22.218210 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.214088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.30.160.1:37925)
0808 02:26:22.214555 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.214560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.214570 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:22.214609 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.214611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.214625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.214723 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:22.214821 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.215503 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.216880 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.217624 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217696 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.217783 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.218068 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.218115 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":109,"threads_started":1}
I20260808 02:26:22.218185 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.213630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43395 (local address 127.0.0.1:47012)
0808 02:26:22.213800 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.213810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.213850 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.214443 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.214444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.214446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:22.214479 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.214481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.215719 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.215720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.216980 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.216983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.217835 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.217896 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.217900 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.217955 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.218022 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":29,"thread_start_us":108,"threads_started":1}
I20260808 02:26:22.218451 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.213630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:37925 (local address 127.0.0.1:39286)
0808 02:26:22.213936 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.213943 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.213980 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.214722 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.214723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.214725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:22.214763 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.214765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.215589 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.215590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.216778 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.216782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.217823 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.217885 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.217889 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.217939 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.218014 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":77,"thread_start_us":110,"threads_started":1}
I20260808 02:26:22.218556 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.213595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:37501 (local address 127.0.0.1:33494)
0808 02:26:22.213717 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.213761 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.213830 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.214140 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.214143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.214145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:22.214212 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.214214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.215598 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.215599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.216980 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.216983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.217682 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.217683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.217760 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.217774 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.218406 (+ 632us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.218451 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":93,"thread_start_us":61,"threads_started":1}
I20260808 02:26:22.219113 17554 tablet_service.cc:1467] Processing CreateTablet for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 (DEFAULT_TABLE table=test-table [id=d81eecc06a724e55b0db027e077dc166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:22.219218 17416 tablet_service.cc:1467] Processing CreateTablet for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 (DEFAULT_TABLE table=test-table [id=d81eecc06a724e55b0db027e077dc166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:22.219496 17554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:22.219527 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:22.219424 17694 tablet_service.cc:1467] Processing CreateTablet for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 (DEFAULT_TABLE table=test-table [id=d81eecc06a724e55b0db027e077dc166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:22.219748 17694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:22.222301 17776 tablet_bootstrap.cc:492] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: Bootstrap starting.
I20260808 02:26:22.222842 17777 tablet_bootstrap.cc:492] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Bootstrap starting.
I20260808 02:26:22.223166 17776 tablet_bootstrap.cc:654] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:22.223498 17777 tablet_bootstrap.cc:654] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:22.223502 17776 log.cc:824] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:22.223836 17777 log.cc:824] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:22.224104 17779 tablet_bootstrap.cc:492] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Bootstrap starting.
I20260808 02:26:22.224393 17776 tablet_bootstrap.cc:492] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: No bootstrap required, opened a new log
I20260808 02:26:22.224484 17776 ts_tablet_manager.cc:1397] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:22.224627 17777 tablet_bootstrap.cc:492] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: No bootstrap required, opened a new log
I20260808 02:26:22.224687 17777 ts_tablet_manager.cc:1397] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:22.224786 17779 tablet_bootstrap.cc:654] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:22.225119 17779 log.cc:824] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:22.225806 17779 tablet_bootstrap.cc:492] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: No bootstrap required, opened a new log
I20260808 02:26:22.225885 17779 ts_tablet_manager.cc:1397] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:22.227313 17776 raft_consensus.cc:348] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.227313 17777 raft_consensus.cc:348] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.227483 17777 raft_consensus.cc:374] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:22.227483 17776 raft_consensus.cc:374] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:22.227522 17777 raft_consensus.cc:729] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6182176ba8ad4d5c89fa40ee1e825da8, State: Initialized, Role: FOLLOWER
I20260808 02:26:22.227522 17776 raft_consensus.cc:729] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57aec456d81e418d97a41c970c38c6a9, State: Initialized, Role: FOLLOWER
I20260808 02:26:22.227664 17777 consensus_queue.cc:260] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.227664 17776 consensus_queue.cc:260] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.227958 17776 ts_tablet_manager.cc:1428] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:26:22.227958 17777 ts_tablet_manager.cc:1428] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:26:22.227929 17779 raft_consensus.cc:348] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.228085 17623 heartbeater.cc:503] Master 127.30.160.62:34987 was elected leader, sending a full tablet report...
I20260808 02:26:22.228087 17485 heartbeater.cc:503] Master 127.30.160.62:34987 was elected leader, sending a full tablet report...
I20260808 02:26:22.228112 17779 raft_consensus.cc:374] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:22.228159 17779 raft_consensus.cc:729] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae077be39034ff7b8acef1cdba607e7, State: Initialized, Role: FOLLOWER
I20260808 02:26:22.228277 17779 consensus_queue.cc:260] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.228535 17779 ts_tablet_manager.cc:1428] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:26:22.228667 17761 heartbeater.cc:503] Master 127.30.160.62:34987 was elected leader, sending a full tablet report...
W20260808 02:26:22.294832 17624 tablet.cc:2369] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:26:22.406143 17486 tablet.cc:2369] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:22.435148 17785 raft_consensus.cc:482] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:22.435271 17785 raft_consensus.cc:504] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.435739 17785 leader_election.cc:288] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57aec456d81e418d97a41c970c38c6a9 (127.30.160.1:37925), 6182176ba8ad4d5c89fa40ee1e825da8 (127.30.160.2:43395)
I20260808 02:26:22.438544 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.436008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:56283 (local address 127.30.160.1:37925)
0808 02:26:22.436057 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.436060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.436072 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:26:22.436111 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.436112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.436123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.436215 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:22.436378 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.436858 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.437776 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.438195 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.438266 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.438276 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.438419 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.438463 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:26:22.438560 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.435939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:37925 (local address 127.30.160.3:56283)
0808 02:26:22.435999 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.436010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.436057 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.436220 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.436222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.436223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:22.436303 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.436305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.436953 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.436954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.437681 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.437683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.438271 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.438272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.438351 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.438355 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.438411 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.438452 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:26:22.438867 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.436046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:43405 (local address 127.30.160.2:43395)
0808 02:26:22.436099 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.436101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.436228 (+ 127us) server_negotiation.cc:407] Connection header received
0808 02:26:22.436291 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.436292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.436305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.436365 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:22.436516 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.436987 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.437916 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.438453 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.438517 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.438524 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.438739 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.438781 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:26:22.438911 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caf38b3c92c4c7c8ffbfae7904b9d79" candidate_uuid: "0ae077be39034ff7b8acef1cdba607e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57aec456d81e418d97a41c970c38c6a9" is_pre_election: true
I20260808 02:26:22.439095 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.435975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:43395 (local address 127.30.160.3:43405)
0808 02:26:22.436170 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.436177 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:22.436237 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:22.436368 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:22.436369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:22.436370 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:22.436440 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.436444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.437073 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.437074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.437823 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:22.437825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.438564 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.438565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:22.438649 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.438653 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:22.438971 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 02:26:22.439020 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":122,"threads_started":1}
I20260808 02:26:22.439095 17437 raft_consensus.cc:2454] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae077be39034ff7b8acef1cdba607e7 in term 0.
I20260808 02:26:22.439371 17646 leader_election.cc:302] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae077be39034ff7b8acef1cdba607e7, 57aec456d81e418d97a41c970c38c6a9; no voters:
I20260808 02:26:22.439461 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caf38b3c92c4c7c8ffbfae7904b9d79" candidate_uuid: "0ae077be39034ff7b8acef1cdba607e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" is_pre_election: true
I20260808 02:26:22.439522 17785 raft_consensus.cc:2781] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 02:26:22.439551 17762 tablet.cc:2369] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:22.439568 17785 raft_consensus.cc:482] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:22.439620 17785 raft_consensus.cc:3037] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:22.439608 17576 raft_consensus.cc:2454] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae077be39034ff7b8acef1cdba607e7 in term 0.
I20260808 02:26:22.440443 17785 raft_consensus.cc:504] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.440591 17785 leader_election.cc:288] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [CANDIDATE]: Term 1 election: Requested vote from peers 57aec456d81e418d97a41c970c38c6a9 (127.30.160.1:37925), 6182176ba8ad4d5c89fa40ee1e825da8 (127.30.160.2:43395)
I20260808 02:26:22.440721 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caf38b3c92c4c7c8ffbfae7904b9d79" candidate_uuid: "0ae077be39034ff7b8acef1cdba607e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57aec456d81e418d97a41c970c38c6a9"
I20260808 02:26:22.440752 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caf38b3c92c4c7c8ffbfae7904b9d79" candidate_uuid: "0ae077be39034ff7b8acef1cdba607e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6182176ba8ad4d5c89fa40ee1e825da8"
I20260808 02:26:22.440807 17437 raft_consensus.cc:3037] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:22.440835 17576 raft_consensus.cc:3037] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:22.441565 17437 raft_consensus.cc:2454] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae077be39034ff7b8acef1cdba607e7 in term 1.
I20260808 02:26:22.441628 17576 raft_consensus.cc:2454] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae077be39034ff7b8acef1cdba607e7 in term 1.
I20260808 02:26:22.441730 17646 leader_election.cc:302] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae077be39034ff7b8acef1cdba607e7, 57aec456d81e418d97a41c970c38c6a9; no voters:
I20260808 02:26:22.441813 17785 raft_consensus.cc:2781] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:22.441943 17785 raft_consensus.cc:686] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 1 LEADER]: Becoming Leader. State: Replica: 0ae077be39034ff7b8acef1cdba607e7, State: Running, Role: LEADER
I20260808 02:26:22.442039 17785 consensus_queue.cc:237] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } }
I20260808 02:26:22.442834 17288 catalog_manager.cc:5495] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ae077be39034ff7b8acef1cdba607e7 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "0ae077be39034ff7b8acef1cdba607e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae077be39034ff7b8acef1cdba607e7" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 } health_report { overall_health: UNKNOWN } } }
I20260808 02:26:22.498987 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.496552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.30.160.3:37501)
0808 02:26:22.496615 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:22.496617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:22.496625 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:22.496670 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:22.496670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:22.496698 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:22.496781 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:22.496919 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.497397 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.498133 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:22.498373 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:22.498451 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:22.498622 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:22.498728 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:22.498832 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:22.498852 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 02:26:22.498874 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:26:22.500865 17576 raft_consensus.cc:1261] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 1 FOLLOWER]: Refusing update from remote peer 0ae077be39034ff7b8acef1cdba607e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:22.500867 17437 raft_consensus.cc:1261] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 57aec456d81e418d97a41c970c38c6a9 [term 1 FOLLOWER]: Refusing update from remote peer 0ae077be39034ff7b8acef1cdba607e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:22.501225 17785 consensus_queue.cc:1035] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6182176ba8ad4d5c89fa40ee1e825da8" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:22.501377 17787 consensus_queue.cc:1035] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57aec456d81e418d97a41c970c38c6a9" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:22.503821 17790 mvcc.cc:205] Tried to move back new op lower bound from 7316094904320610304 to 7316094904083103744. Current Snapshot: MvccSnapshot[applied={T|T < 7316094904320610304}]
I20260808 02:26:22.504382 17792 mvcc.cc:205] Tried to move back new op lower bound from 7316094904320610304 to 7316094904083103744. Current Snapshot: MvccSnapshot[applied={T|T < 7316094904320610304}]
I20260808 02:26:22.505992 17791 mvcc.cc:205] Tried to move back new op lower bound from 7316094904320610304 to 7316094904083103744. Current Snapshot: MvccSnapshot[applied={T|T < 7316094904320610304 or (T in {7316094904320610304})}]
I20260808 02:26:22.506212 17290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:22.506305 17290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:22.507025 17290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76a2b739d56f49d2ae37b3d69df44421: Sending DeleteTablet for 3 replicas of tablet 9caf38b3c92c4c7c8ffbfae7904b9d79
I20260808 02:26:22.507311 17694 tablet_service.cc:1514] Processing DeleteTablet for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:22 UTC) from {username='slave'} at 127.0.0.1:33494
I20260808 02:26:22.507332 17416 tablet_service.cc:1514] Processing DeleteTablet for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:22 UTC) from {username='slave'} at 127.0.0.1:39286
I20260808 02:26:22.507380 17554 tablet_service.cc:1514] Processing DeleteTablet for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:22 UTC) from {username='slave'} at 127.0.0.1:47012
I20260808 02:26:22.507634 17801 tablet_replica.cc:330] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: stopping tablet replica
I20260808 02:26:22.507715 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17353
I20260808 02:26:22.508054 17801 raft_consensus.cc:2229] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:26:22.508208 17801 raft_consensus.cc:2258] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:22.508714 17801 ts_tablet_manager.cc:1905] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:22.509048 17802 tablet_replica.cc:330] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: stopping tablet replica
I20260808 02:26:22.509174 17802 raft_consensus.cc:2229] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:22.509317 17802 raft_consensus.cc:2258] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:22.509960 17802 ts_tablet_manager.cc:1905] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:22.510941 17801 ts_tablet_manager.cc:1918] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:22.511062 17801 log.cc:1196] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/9caf38b3c92c4c7c8ffbfae7904b9d79
I20260808 02:26:22.511439 17801 ts_tablet_manager.cc:1939] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 0ae077be39034ff7b8acef1cdba607e7: Deleting consensus metadata
I20260808 02:26:22.511914 17273 catalog_manager.cc:4841] TS 0ae077be39034ff7b8acef1cdba607e7 (127.30.160.3:37501): tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 (table test-table [id=d81eecc06a724e55b0db027e077dc166]) successfully deleted
I20260808 02:26:22.512245 17802 ts_tablet_manager.cc:1918] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:22.512339 17802 log.cc:1196] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/9caf38b3c92c4c7c8ffbfae7904b9d79
I20260808 02:26:22.512661 17802 ts_tablet_manager.cc:1939] T 9caf38b3c92c4c7c8ffbfae7904b9d79 P 6182176ba8ad4d5c89fa40ee1e825da8: Deleting consensus metadata
I20260808 02:26:22.516865 17275 catalog_manager.cc:4841] TS 6182176ba8ad4d5c89fa40ee1e825da8 (127.30.160.2:43395): tablet 9caf38b3c92c4c7c8ffbfae7904b9d79 (table test-table [id=d81eecc06a724e55b0db027e077dc166]) successfully deleted
W20260808 02:26:22.522104 17274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:37925 (error 108)
I20260808 02:26:22.522337 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17488
I20260808 02:26:22.523216 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:22.523034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:37925 (local address 127.0.0.1:39294)
0808 02:26:22.523101 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:22.523182 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:37925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:26:22.524184 17274 catalog_manager.cc:4584] TS 57aec456d81e418d97a41c970c38c6a9 (127.30.160.1:37925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9caf38b3c92c4c7c8ffbfae7904b9d79: Network error: Client connection negotiation failed: client connection to 127.30.160.1:37925: connect: Connection refused (error 111)
I20260808 02:26:22.528573 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17626
I20260808 02:26:22.534751 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17257
2026-08-08T02:26:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (932 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:22 dist-test-slave-zkpd krb5kdc[17807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:22 dist-test-slave-zkpd krb5kdc[17807](info): set up 2 sockets
Aug 08 02:26:22 dist-test-slave-zkpd krb5kdc[17807](info): commencing operation
krb5kdc: starting...
W20260808 02:26:23.587599 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:23 dist-test-slave-zkpd krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:23.704879 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38081
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:38081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:23.785609 17820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:23.785830 17820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:23.785868 17820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:23.786988 17820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:23.787050 17820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:23.787074 17820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:23.787106 17820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:23.787132 17820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:23.788686 17820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:38081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:23.788954 17820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:23.789176 17820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:23.793957 17825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:23.793956 17828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:23.793996 17826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:23.794024 17820 server_base.cc:1034] running on GCE node
I20260808 02:26:23.794411 17820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:23.794704 17820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:23.795868 17820 hybrid_clock.cc:648] HybridClock initialized: now 1786155983795857 us; error 36 us; skew 500 ppm
Aug 08 02:26:23 dist-test-slave-zkpd krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155983, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:23.798307 17820 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:23.798678 17820 webserver.cc:460] Webserver started at http://127.30.160.62:38673/ using document root <none> and password file <none>
I20260808 02:26:23.798883 17820 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:23.798940 17820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:23.799093 17820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:23.800185 17820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "6ff56ef0e5e64097b29da3ce929c6ed3"
format_stamp: "Formatted at 2026-08-08 02:26:23 on dist-test-slave-zkpd"
server_key: "a8adc49ec495c5dc17eb15387981865f"
server_key_iv: "77d8d2adbb026f2c8f31d9f02b7c1415"
server_key_version: "encryptionkey@0"
I20260808 02:26:23.800549 17820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "6ff56ef0e5e64097b29da3ce929c6ed3"
format_stamp: "Formatted at 2026-08-08 02:26:23 on dist-test-slave-zkpd"
server_key: "a8adc49ec495c5dc17eb15387981865f"
server_key_iv: "77d8d2adbb026f2c8f31d9f02b7c1415"
server_key_version: "encryptionkey@0"
I20260808 02:26:23.801864 17820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:23.802567 17835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:23.802752 17820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:23.802831 17820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "6ff56ef0e5e64097b29da3ce929c6ed3"
format_stamp: "Formatted at 2026-08-08 02:26:23 on dist-test-slave-zkpd"
server_key: "a8adc49ec495c5dc17eb15387981865f"
server_key_iv: "77d8d2adbb026f2c8f31d9f02b7c1415"
server_key_version: "encryptionkey@0"
I20260808 02:26:23.802884 17820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:23.829821 17820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:23.832104 17820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:23.832229 17820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:23.836130 17820 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:38081
I20260808 02:26:23.836211 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:38081 every 8 connection(s)
I20260808 02:26:23.836524 17820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:23.837376 17890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:23.839643 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17820
I20260808 02:26:23.839669 17890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: Bootstrap starting.
I20260808 02:26:23.839794 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:23.840042 31360 external_mini_cluster.cc:1428] Setting key 8287eeb4eebfeff63dc13f1253abac75
I20260808 02:26:23.840402 17890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:23.840731 17890 log.cc:824] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:23.841476 17890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: No bootstrap required, opened a new log
I20260808 02:26:23.843683 17890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38081 } }
I20260808 02:26:23.843844 17890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:23.843875 17890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff56ef0e5e64097b29da3ce929c6ed3, State: Initialized, Role: FOLLOWER
I20260808 02:26:23.844031 17890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38081 } }
I20260808 02:26:23.844110 17890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:23.844151 17890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:23.844211 17890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:23.844774 17890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38081 } }
I20260808 02:26:23.844897 17890 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff56ef0e5e64097b29da3ce929c6ed3; no voters:
I20260808 02:26:23.845063 17890 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:23.845093 17895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:23.845228 17895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff56ef0e5e64097b29da3ce929c6ed3, State: Running, Role: LEADER
I20260808 02:26:23.845350 17890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:23.845379 17895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38081 } }
Aug 08 02:26:23 dist-test-slave-zkpd krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:23.847882 17896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff56ef0e5e64097b29da3ce929c6ed3. Latest consensus state: current_term: 1 leader_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38081 } } }
I20260808 02:26:23.848264 17896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:23.848048 17897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff56ef0e5e64097b29da3ce929c6ed3" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38081 } } }
I20260808 02:26:23.848551 17897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3 [sys.catalog]: This master's current role is: LEADER
W20260808 02:26:23.849030 17910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:23.849089 17910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:23.849146 17912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:23.849327 17912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:23.850914 17912 catalog_manager.cc:1347] Generated new cluster ID: 11243424adb74353859f8d470aeebf21
I20260808 02:26:23.850960 17912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:23.851168 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:23.840568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.30.160.62:38081)
0808 02:26:23.840750 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:23.840758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:23.840789 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:26:23.841246 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:23.841252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:23.841296 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:23.841409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:23.842182 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:23.843042 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:23.844166 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:23.844632 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:23.844714 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:23.847510 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:23.847514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:23.847515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:23.847528 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:23.849544 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:23.849912 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:23.849915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:23.849915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:23.850013 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:23.850230 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:23.850231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:23.850232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:23.850544 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:23.850703 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:23.850828 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 02:26:23.850973 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260808 02:26:23.876750 17912 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:23.877393 17912 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:23.883013 17912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: Generated new TSK 0
I20260808 02:26:23.883204 17912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:26:23.911497 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38081
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:23.993657 17917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:23.993870 17917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:23.993892 17917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:23.994940 17917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:23.994973 17917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:23.994998 17917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:23.996423 17917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:23.996604 17917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:23.996768 17917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:26:24.001632 17917 server_base.cc:1034] running on GCE node
W20260808 02:26:24.001603 17922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.001614 17925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.001611 17923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.002032 17917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.002254 17917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.003386 17917 hybrid_clock.cc:648] HybridClock initialized: now 1786155984003374 us; error 27 us; skew 500 ppm
Aug 08 02:26:24 dist-test-slave-zkpd krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:24.006235 17917 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:26:24.006693 17917 webserver.cc:460] Webserver started at http://127.30.160.1:42723/ using document root <none> and password file <none>
I20260808 02:26:24.006930 17917 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.006974 17917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.007120 17917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.008203 17917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "ce71ca08c7d34c13ab8a804e8cf9a089"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "25a1f14f376fe1152b28df056a392920"
server_key_iv: "c25a548776441b6d4bf84a4a8dd66240"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.008560 17917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "ce71ca08c7d34c13ab8a804e8cf9a089"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "25a1f14f376fe1152b28df056a392920"
server_key_iv: "c25a548776441b6d4bf84a4a8dd66240"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.009871 17917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 02:26:24.010617 17932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.010797 17917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:24.010891 17917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "ce71ca08c7d34c13ab8a804e8cf9a089"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "25a1f14f376fe1152b28df056a392920"
server_key_iv: "c25a548776441b6d4bf84a4a8dd66240"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.010946 17917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:24.018968 17917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:24.021145 17917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:24.021250 17917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:24.021422 17917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:24.021659 17917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:24.021688 17917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.021708 17917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:24.021720 17917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.027508 17917 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:46215
I20260808 02:26:24.027593 18051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:46215 every 8 connection(s)
I20260808 02:26:24.027875 17917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:26:24 dist-test-slave-zkpd krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:24.036736 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 17917
I20260808 02:26:24.036872 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:24.036863 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.028607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38081 (local address 127.30.160.1:34207)
0808 02:26:24.028813 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.028849 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.029302 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.029503 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.029506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.030104 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:24.030284 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.030291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.030965 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.030965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.031738 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.031740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.032159 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.032159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.032259 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.032693 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:24.032700 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:24.034029 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:24.035616 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:24.035622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:24.035624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:24.035837 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:24.036049 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:24.036050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:24.036050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:24.036147 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:24.036360 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:24.036365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:24.036472 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.036551 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.036665 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":115,"threads_started":1}
I20260808 02:26:24.036926 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.028651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:34207 (local address 127.30.160.62:38081)
0808 02:26:24.028749 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.028752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.028934 (+ 182us) server_negotiation.cc:407] Connection header received
0808 02:26:24.029382 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.029383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.029392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.029451 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.030378 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.030878 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.031817 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.032086 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.032139 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.034114 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.034116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.034116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:24.034123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.035547 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.035933 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.035934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.035935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.035991 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.036204 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.036205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.036205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.036306 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.036354 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.036704 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.036796 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:26:24.037084 31360 external_mini_cluster.cc:1428] Setting key 0f8bdb651d45cb3f0102f52f4013030a
I20260808 02:26:24.037737 18053 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38081
I20260808 02:26:24.037889 18053 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:24.038069 18053 heartbeater.cc:511] Master 127.30.160.62:38081 requested a full tablet report, sending...
I20260808 02:26:24.038616 17852 ts_manager.cc:194] Registered new tserver with Master: ce71ca08c7d34c13ab8a804e8cf9a089 (127.30.160.1:46215)
I20260808 02:26:24.039222 17852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:34207
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:26:24.062564 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38081
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:24.146008 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:24.146227 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:24.146251 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:24.147513 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:24.147554 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:24.147580 18058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:24.149032 18058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:24.149225 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:24.149395 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:24.154157 18064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.154165 18065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.154145 18067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.154397 18058 server_base.cc:1034] running on GCE node
I20260808 02:26:24.154587 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.154831 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.155958 18058 hybrid_clock.cc:648] HybridClock initialized: now 1786155984155920 us; error 43 us; skew 500 ppm
Aug 08 02:26:24 dist-test-slave-zkpd krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:24.158432 18058 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:26:24.158886 18058 webserver.cc:460] Webserver started at http://127.30.160.2:40949/ using document root <none> and password file <none>
I20260808 02:26:24.159094 18058 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.159137 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.159302 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.160385 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "e1f8da179cb34703bf5cf512eb448218"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "5b8b904bfb984f566696e3901207a7b8"
server_key_iv: "59b80b6bfd10193f34679a8aff024576"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.160746 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "e1f8da179cb34703bf5cf512eb448218"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "5b8b904bfb984f566696e3901207a7b8"
server_key_iv: "59b80b6bfd10193f34679a8aff024576"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.162050 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:24.162768 18074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.162940 18058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:24.163010 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "e1f8da179cb34703bf5cf512eb448218"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "5b8b904bfb984f566696e3901207a7b8"
server_key_iv: "59b80b6bfd10193f34679a8aff024576"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.163100 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:24.176373 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:24.178212 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:24.178368 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:24.178582 18058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:24.178898 18058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:24.178954 18058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.178997 18058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:24.179025 18058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.184729 18058 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:44929
I20260808 02:26:24.184775 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:44929 every 8 connection(s)
I20260808 02:26:24.185072 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:24.187685 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18058
I20260808 02:26:24.187811 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:24.188027 31360 external_mini_cluster.cc:1428] Setting key 71a1ba61d1b2657c4cbcc9ba382d8d92
Aug 08 02:26:24 dist-test-slave-zkpd krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:24.194787 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.185770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:58689 (local address 127.30.160.62:38081)
0808 02:26:24.185832 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.185834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.186014 (+ 180us) server_negotiation.cc:407] Connection header received
0808 02:26:24.186512 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.186513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.186527 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.186580 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.187469 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.188163 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.189239 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.189448 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.189523 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.191962 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.191964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.191964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:24.191971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.193424 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.193826 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.193828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.193828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.193891 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.194150 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.194151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.194151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.194265 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.194343 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.194616 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.194699 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:26:24.194960 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.185744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38081 (local address 127.30.160.2:58689)
0808 02:26:24.185900 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.185932 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.186398 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.186651 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.186654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.187188 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:24.187369 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.187373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.188254 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.188255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.189140 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.189143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.189530 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.189532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.189654 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.190151 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:24.190159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:24.191837 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:24.193515 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:24.193522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:24.193525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:24.193749 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:24.193960 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:24.193961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:24.193962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:24.194080 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:24.194334 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:24.194341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:24.194480 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.194580 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.194730 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260808 02:26:24.195433 18189 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38081
I20260808 02:26:24.195520 18189 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:24.195693 18189 heartbeater.cc:511] Master 127.30.160.62:38081 requested a full tablet report, sending...
I20260808 02:26:24.196087 17852 ts_manager.cc:194] Registered new tserver with Master: e1f8da179cb34703bf5cf512eb448218 (127.30.160.2:44929)
I20260808 02:26:24.196468 17852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:58689
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:26:24.214654 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38081
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:24.295600 18194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:24.295828 18194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:24.295869 18194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:24.297003 18194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:24.297070 18194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:24.297115 18194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:24.298578 18194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:24.298842 18194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:24.299072 18194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:24.303956 18200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.303973 18203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.303973 18201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.304214 18194 server_base.cc:1034] running on GCE node
I20260808 02:26:24.304414 18194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.304649 18194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.305866 18194 hybrid_clock.cc:648] HybridClock initialized: now 1786155984305840 us; error 39 us; skew 500 ppm
Aug 08 02:26:24 dist-test-slave-zkpd krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:24.308396 18194 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:26:24.308772 18194 webserver.cc:460] Webserver started at http://127.30.160.3:41305/ using document root <none> and password file <none>
I20260808 02:26:24.309001 18194 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.309067 18194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.309170 18194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.310096 18194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "ccbcc31477f9439abbe007d7e79e1b50"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "1be1318dd004b6e5950454c56389a53a"
server_key_iv: "bc15278b74bcb8f747edb71616b4bb98"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.310408 18194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "ccbcc31477f9439abbe007d7e79e1b50"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "1be1318dd004b6e5950454c56389a53a"
server_key_iv: "bc15278b74bcb8f747edb71616b4bb98"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.311631 18194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:24.312386 18210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.312577 18194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.312654 18194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "ccbcc31477f9439abbe007d7e79e1b50"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "1be1318dd004b6e5950454c56389a53a"
server_key_iv: "bc15278b74bcb8f747edb71616b4bb98"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.312707 18194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:24.320981 18194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:24.323094 18194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:24.323196 18194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:24.323356 18194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:24.323645 18194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:24.323679 18194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.323700 18194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:24.323746 18194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.329607 18194 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:46457
I20260808 02:26:24.329738 18325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:46457 every 8 connection(s)
I20260808 02:26:24.329989 18194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:26:24 dist-test-slave-zkpd krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:24.339005 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.330711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:55409 (local address 127.30.160.62:38081)
0808 02:26:24.330753 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.330755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.330998 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:26:24.331514 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.331515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.331527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.331580 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.332563 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.333098 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.334094 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.334294 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.334379 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.336530 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.336532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.336532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:24.336538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.337927 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.338258 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.338259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.338260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.338307 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.338508 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.338508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.338509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.338602 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.338651 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.338874 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.338940 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:26:24.339147 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.330688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38081 (local address 127.30.160.3:55409)
0808 02:26:24.330881 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.330914 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.331392 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.331656 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.331662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.332277 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:24.332465 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.332470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.333196 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.333197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.333990 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.333992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.334391 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.334392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.334533 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.335016 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:24.335024 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:24.336447 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:24.337993 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:24.337999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:24.338000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:24.338197 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:24.338363 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:24.338364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:24.338364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:24.338454 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:24.338654 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:24.338659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:24.338761 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.338850 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.338962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260808 02:26:24.339524 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18194
I20260808 02:26:24.339658 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:24.339640 18326 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38081
I20260808 02:26:24.339744 18326 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:24.339923 31360 external_mini_cluster.cc:1428] Setting key 31cb1ba7fa2e9ccfbf2e7eef49a38f10
I20260808 02:26:24.339967 18326 heartbeater.cc:511] Master 127.30.160.62:38081 requested a full tablet report, sending...
I20260808 02:26:24.340339 17851 ts_manager.cc:194] Registered new tserver with Master: ccbcc31477f9439abbe007d7e79e1b50 (127.30.160.3:46457)
I20260808 02:26:24.340857 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:24.340850 17851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:55409
I20260808 02:26:24.348013 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.342221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.30.160.62:38081)
0808 02:26:24.342259 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.342261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.342411 (+ 150us) server_negotiation.cc:407] Connection header received
0808 02:26:24.342446 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.342446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.342454 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.342493 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.343004 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.343431 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.344174 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.344336 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.344383 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.345511 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.345512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.345512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:24.345518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.346807 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.347146 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.347146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.347147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.347194 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:24.347380 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:24.347381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:24.347381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:24.347528 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.347611 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.347814 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.347863 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 02:26:24.349617 17851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:24.351115 17851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:24.357388 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.353445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:44929 (local address 127.0.0.1:50144)
0808 02:26:24.353654 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.353664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.353727 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.354054 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.354055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.354057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:24.354092 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.354093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.355125 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.355126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.356193 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.356196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.357086 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.357086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.357163 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.357176 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.357264 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.357315 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":3,"thread_start_us":34,"threads_started":1}
I20260808 02:26:24.357396 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.353580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.30.160.2:44929)
0808 02:26:24.353776 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.353781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.353792 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:24.353832 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.353834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.353848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.353969 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:24.354165 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.355022 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.356289 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.357005 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.357087 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.357145 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.357284 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.357323 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260808 02:26:24.357590 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.353223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:46215 (local address 127.0.0.1:42096)
0808 02:26:24.353354 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.353384 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.353455 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.353996 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.354000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.354002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:24.354073 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.354075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.355521 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.355522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.356547 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.356550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.357364 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.357365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.357441 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.357446 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.357477 (+ 31us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.357515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":8,"thread_start_us":66,"threads_started":1}
I20260808 02:26:24.357904 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.353557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:46457 (local address 127.0.0.1:44068)
0808 02:26:24.354085 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.354094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.354136 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.354414 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.354415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.354418 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:24.354451 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.354453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.355663 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.355664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.356792 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.356795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.357658 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.357660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.357728 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.357732 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.357800 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.357837 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260808 02:26:24.358036 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.353351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.30.160.1:46215)
0808 02:26:24.353823 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.353830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.353844 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:26:24.353877 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.353878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.353890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.353983 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:24.354268 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.355412 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.356626 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.357262 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.357820 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.357885 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.357909 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.357948 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":86,"threads_started":1}
I20260808 02:26:24.358107 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.353656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.30.160.3:46457)
0808 02:26:24.354243 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.354248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.354256 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:24.354299 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.354301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.354317 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.354519 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:24.354580 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.355565 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.356875 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.357578 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.357653 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.357713 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.357989 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.358025 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":101,"threads_started":1}
I20260808 02:26:24.358402 18122 tablet_service.cc:1467] Processing CreateTablet for tablet c39aec2a757a45a09a2a3ed4c1278538 (DEFAULT_TABLE table=test-table [id=9c3766127fb94b0b9873fa1c646554c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:24.358758 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39aec2a757a45a09a2a3ed4c1278538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:24.358743 17983 tablet_service.cc:1467] Processing CreateTablet for tablet c39aec2a757a45a09a2a3ed4c1278538 (DEFAULT_TABLE table=test-table [id=9c3766127fb94b0b9873fa1c646554c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:24.359078 17983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39aec2a757a45a09a2a3ed4c1278538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:24.360612 18258 tablet_service.cc:1467] Processing CreateTablet for tablet c39aec2a757a45a09a2a3ed4c1278538 (DEFAULT_TABLE table=test-table [id=9c3766127fb94b0b9873fa1c646554c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:24.360939 18258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39aec2a757a45a09a2a3ed4c1278538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:24.361891 18342 tablet_bootstrap.cc:492] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Bootstrap starting.
I20260808 02:26:24.362221 18343 tablet_bootstrap.cc:492] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089: Bootstrap starting.
I20260808 02:26:24.362629 18342 tablet_bootstrap.cc:654] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:24.362869 18343 tablet_bootstrap.cc:654] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:24.363253 18342 log.cc:824] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:24.363493 18343 log.cc:824] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:24.364069 18342 tablet_bootstrap.cc:492] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: No bootstrap required, opened a new log
I20260808 02:26:24.364200 18342 ts_tablet_manager.cc:1397] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:24.364280 18343 tablet_bootstrap.cc:492] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089: No bootstrap required, opened a new log
I20260808 02:26:24.364391 18343 ts_tablet_manager.cc:1397] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:24.365664 18342 raft_consensus.cc:348] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.365866 18342 raft_consensus.cc:374] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:24.365914 18342 raft_consensus.cc:729] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f8da179cb34703bf5cf512eb448218, State: Initialized, Role: FOLLOWER
I20260808 02:26:24.366029 18342 consensus_queue.cc:260] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.366056 18343 raft_consensus.cc:348] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.366235 18343 raft_consensus.cc:374] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:24.366271 18343 raft_consensus.cc:729] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce71ca08c7d34c13ab8a804e8cf9a089, State: Initialized, Role: FOLLOWER
I20260808 02:26:24.366307 18342 ts_tablet_manager.cc:1428] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:24.366323 18189 heartbeater.cc:503] Master 127.30.160.62:38081 was elected leader, sending a full tablet report...
I20260808 02:26:24.366387 18343 consensus_queue.cc:260] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.366657 18343 ts_tablet_manager.cc:1428] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:26:24.366735 18053 heartbeater.cc:503] Master 127.30.160.62:38081 was elected leader, sending a full tablet report...
I20260808 02:26:24.366899 18346 tablet_bootstrap.cc:492] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Bootstrap starting.
I20260808 02:26:24.367609 18346 tablet_bootstrap.cc:654] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:24.368152 18346 log.cc:824] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:24.368763 18346 tablet_bootstrap.cc:492] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: No bootstrap required, opened a new log
I20260808 02:26:24.368816 18346 ts_tablet_manager.cc:1397] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:24.370272 18346 raft_consensus.cc:348] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.370404 18346 raft_consensus.cc:374] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:24.370425 18346 raft_consensus.cc:729] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbcc31477f9439abbe007d7e79e1b50, State: Initialized, Role: FOLLOWER
I20260808 02:26:24.370527 18346 consensus_queue.cc:260] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.370759 18346 ts_tablet_manager.cc:1428] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:24.370810 18326 heartbeater.cc:503] Master 127.30.160.62:38081 was elected leader, sending a full tablet report...
I20260808 02:26:24.405886 18350 raft_consensus.cc:482] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:24.406014 18350 raft_consensus.cc:504] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.406366 18350 leader_election.cc:288] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce71ca08c7d34c13ab8a804e8cf9a089 (127.30.160.1:46215), e1f8da179cb34703bf5cf512eb448218 (127.30.160.2:44929)
I20260808 02:26:24.409319 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.406559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:46215 (local address 127.30.160.3:40443)
0808 02:26:24.406708 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.406722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.406781 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.406968 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.406969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.406971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:24.407046 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.407048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.407693 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.407695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.408496 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.408498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.409058 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.409059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.409141 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.409144 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.409210 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.409248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1}
I20260808 02:26:24.409318 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.406639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:40443 (local address 127.30.160.1:46215)
0808 02:26:24.406711 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.406713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.406781 (+ 68us) server_negotiation.cc:407] Connection header received
0808 02:26:24.406886 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.406886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.406894 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.406965 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:24.407120 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.407611 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.408588 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.408979 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.409062 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.409070 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.409217 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.409254 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:26:24.409426 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.406792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:44719 (local address 127.30.160.2:44929)
0808 02:26:24.406845 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.406847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.406972 (+ 125us) server_negotiation.cc:407] Connection header received
0808 02:26:24.407055 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.407056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.407065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.407106 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:24.407249 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.407709 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.408679 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.409062 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.409147 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.409155 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.409327 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.409364 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:26:24.409446 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.406690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:44929 (local address 127.30.160.3:44719)
0808 02:26:24.406885 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.406895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.406983 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.407119 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.407120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.407121 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:24.407196 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.407198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.407793 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.407794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.408583 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.408585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.409147 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.409148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.409224 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.409230 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.409317 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.409357 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":144,"threads_started":1}
I20260808 02:26:24.409744 18004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39aec2a757a45a09a2a3ed4c1278538" candidate_uuid: "ccbcc31477f9439abbe007d7e79e1b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" is_pre_election: true
I20260808 02:26:24.409789 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39aec2a757a45a09a2a3ed4c1278538" candidate_uuid: "ccbcc31477f9439abbe007d7e79e1b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f8da179cb34703bf5cf512eb448218" is_pre_election: true
I20260808 02:26:24.409906 18004 raft_consensus.cc:2454] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbcc31477f9439abbe007d7e79e1b50 in term 0.
I20260808 02:26:24.409931 18143 raft_consensus.cc:2454] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbcc31477f9439abbe007d7e79e1b50 in term 0.
I20260808 02:26:24.410123 18214 leader_election.cc:302] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbcc31477f9439abbe007d7e79e1b50, ce71ca08c7d34c13ab8a804e8cf9a089; no voters:
I20260808 02:26:24.410234 18350 raft_consensus.cc:2781] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:24.410290 18350 raft_consensus.cc:482] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:24.410322 18350 raft_consensus.cc:3037] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:24.411042 18350 raft_consensus.cc:504] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.411187 18350 leader_election.cc:288] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [CANDIDATE]: Term 1 election: Requested vote from peers ce71ca08c7d34c13ab8a804e8cf9a089 (127.30.160.1:46215), e1f8da179cb34703bf5cf512eb448218 (127.30.160.2:44929)
I20260808 02:26:24.411345 18004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39aec2a757a45a09a2a3ed4c1278538" candidate_uuid: "ccbcc31477f9439abbe007d7e79e1b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089"
I20260808 02:26:24.411361 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39aec2a757a45a09a2a3ed4c1278538" candidate_uuid: "ccbcc31477f9439abbe007d7e79e1b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f8da179cb34703bf5cf512eb448218"
I20260808 02:26:24.411437 18004 raft_consensus.cc:3037] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:24.411434 18143 raft_consensus.cc:3037] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:24.412249 18004 raft_consensus.cc:2454] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbcc31477f9439abbe007d7e79e1b50 in term 1.
I20260808 02:26:24.412257 18143 raft_consensus.cc:2454] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbcc31477f9439abbe007d7e79e1b50 in term 1.
I20260808 02:26:24.412426 18214 leader_election.cc:302] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbcc31477f9439abbe007d7e79e1b50, ce71ca08c7d34c13ab8a804e8cf9a089; no voters:
I20260808 02:26:24.412552 18350 raft_consensus.cc:2781] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:24.412684 18350 raft_consensus.cc:686] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 1 LEADER]: Becoming Leader. State: Replica: ccbcc31477f9439abbe007d7e79e1b50, State: Running, Role: LEADER
I20260808 02:26:24.412792 18350 consensus_queue.cc:237] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } }
I20260808 02:26:24.413610 17851 catalog_manager.cc:5495] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccbcc31477f9439abbe007d7e79e1b50 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "ccbcc31477f9439abbe007d7e79e1b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbcc31477f9439abbe007d7e79e1b50" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20260808 02:26:24.430527 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.428119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.30.160.3:46457)
0808 02:26:24.428162 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.428164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.428173 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:24.428246 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.428246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.428265 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.428331 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:24.428454 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.428940 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.429705 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.429916 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.429991 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.430136 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:24.430229 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:24.430347 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.430410 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.430422 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:26:24.432317 18004 raft_consensus.cc:1261] T c39aec2a757a45a09a2a3ed4c1278538 P ce71ca08c7d34c13ab8a804e8cf9a089 [term 1 FOLLOWER]: Refusing update from remote peer ccbcc31477f9439abbe007d7e79e1b50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:24.432344 18143 raft_consensus.cc:1261] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 1 FOLLOWER]: Refusing update from remote peer ccbcc31477f9439abbe007d7e79e1b50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:24.432691 18350 consensus_queue.cc:1035] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce71ca08c7d34c13ab8a804e8cf9a089" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:24.432842 18352 consensus_queue.cc:1035] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f8da179cb34703bf5cf512eb448218" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:24.435045 18355 mvcc.cc:205] Tried to move back new op lower bound from 7316094912232022016 to 7316094912155459584. Current Snapshot: MvccSnapshot[applied={T|T < 7316094912232022016}]
I20260808 02:26:24.435482 18357 mvcc.cc:205] Tried to move back new op lower bound from 7316094912232022016 to 7316094912155459584. Current Snapshot: MvccSnapshot[applied={T|T < 7316094912232022016 or (T in {7316094912232022016})}]
I20260808 02:26:24.436137 18356 mvcc.cc:205] Tried to move back new op lower bound from 7316094912232022016 to 7316094912155459584. Current Snapshot: MvccSnapshot[applied={T|T < 7316094912232022016 or (T in {7316094912232022016})}]
I20260808 02:26:24.437439 17851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:24.437529 17851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:24.438205 17851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff56ef0e5e64097b29da3ce929c6ed3: Sending DeleteTablet for 3 replicas of tablet c39aec2a757a45a09a2a3ed4c1278538
I20260808 02:26:24.438599 18122 tablet_service.cc:1514] Processing DeleteTablet for tablet c39aec2a757a45a09a2a3ed4c1278538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:24 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:50144
I20260808 02:26:24.438606 17983 tablet_service.cc:1514] Processing DeleteTablet for tablet c39aec2a757a45a09a2a3ed4c1278538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:24 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:42096
I20260808 02:26:24.438777 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17917
I20260808 02:26:24.438866 18258 tablet_service.cc:1514] Processing DeleteTablet for tablet c39aec2a757a45a09a2a3ed4c1278538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:24 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:44068
I20260808 02:26:24.438920 18364 tablet_replica.cc:330] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: stopping tablet replica
I20260808 02:26:24.439013 18364 raft_consensus.cc:2229] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:24.439105 18364 raft_consensus.cc:2258] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:24.439544 18364 ts_tablet_manager.cc:1905] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:24.441311 18364 ts_tablet_manager.cc:1918] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:24.441459 18364 log.cc:1196] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/c39aec2a757a45a09a2a3ed4c1278538
I20260808 02:26:24.441838 18364 ts_tablet_manager.cc:1939] T c39aec2a757a45a09a2a3ed4c1278538 P e1f8da179cb34703bf5cf512eb448218: Deleting consensus metadata
I20260808 02:26:24.442227 18366 tablet_replica.cc:330] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: stopping tablet replica
I20260808 02:26:24.442307 17839 catalog_manager.cc:4841] TS e1f8da179cb34703bf5cf512eb448218 (127.30.160.2:44929): tablet c39aec2a757a45a09a2a3ed4c1278538 (table test-table [id=9c3766127fb94b0b9873fa1c646554c4]) successfully deleted
I20260808 02:26:24.442366 18366 raft_consensus.cc:2229] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:26:24.442456 18366 raft_consensus.cc:2258] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:24.442924 18366 ts_tablet_manager.cc:1905] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:24.444877 18366 ts_tablet_manager.cc:1918] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:24.445001 18366 log.cc:1196] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/c39aec2a757a45a09a2a3ed4c1278538
I20260808 02:26:24.445307 18366 ts_tablet_manager.cc:1939] T c39aec2a757a45a09a2a3ed4c1278538 P ccbcc31477f9439abbe007d7e79e1b50: Deleting consensus metadata
I20260808 02:26:24.445725 17839 catalog_manager.cc:4841] TS ccbcc31477f9439abbe007d7e79e1b50 (127.30.160.3:46457): tablet c39aec2a757a45a09a2a3ed4c1278538 (table test-table [id=9c3766127fb94b0b9873fa1c646554c4]) successfully deleted
W20260808 02:26:24.447160 17839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:46215 (error 108)
I20260808 02:26:24.447394 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18058
I20260808 02:26:24.448143 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.447551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:46215 (local address 127.0.0.1:42100)
0808 02:26:24.448063 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.448104 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
W20260808 02:26:24.449102 17839 catalog_manager.cc:4584] TS ce71ca08c7d34c13ab8a804e8cf9a089 (127.30.160.1:46215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c39aec2a757a45a09a2a3ed4c1278538: Network error: Client connection negotiation failed: client connection to 127.30.160.1:46215: connect: Connection refused (error 111)
I20260808 02:26:24.456331 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18194
I20260808 02:26:24.461854 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 17820
2026-08-08T02:26:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T02:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:24Z Disabled control of system clock
I20260808 02:26:24.490762 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:32817
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:32817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:24.571493 18371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:24.571686 18371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:24.571705 18371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:24.572755 18371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:24.572798 18371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:24.572810 18371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:24.572822 18371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:24.572835 18371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:24.574090 18371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:32817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:32817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:24.574265 18371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:24.574492 18371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:24.579172 18377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.579227 18376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.579228 18379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.579561 18371 server_base.cc:1034] running on GCE node
I20260808 02:26:24.579787 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.580222 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.581398 18371 hybrid_clock.cc:648] HybridClock initialized: now 1786155984581380 us; error 46 us; skew 500 ppm
I20260808 02:26:24.586558 18371 webserver.cc:460] Webserver started at http://127.30.160.62:41437/ using document root <none> and password file <none>
I20260808 02:26:24.586859 18371 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.586941 18371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.587109 18371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.588382 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "613ee17e132d43593035a24caf52dbae"
server_key_iv: "8d9f2205039e763e5b9ff0d672db46c2"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.588775 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "613ee17e132d43593035a24caf52dbae"
server_key_iv: "8d9f2205039e763e5b9ff0d672db46c2"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.590232 18371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:24.591035 18385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.591223 18371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:24.591336 18371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "613ee17e132d43593035a24caf52dbae"
server_key_iv: "8d9f2205039e763e5b9ff0d672db46c2"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.591413 18371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:24.611172 18371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:24.611480 18371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:24.611663 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:24.615427 18371 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:32817
I20260808 02:26:24.615479 18440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:32817 every 8 connection(s)
I20260808 02:26:24.615825 18371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:24.616765 18441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:24.618983 18441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1: Bootstrap starting.
I20260808 02:26:24.619565 18441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:24.619874 18441 log.cc:824] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:24.620568 18441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1: No bootstrap required, opened a new log
I20260808 02:26:24.622093 18441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 32817 } }
I20260808 02:26:24.622224 18441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:24.622246 18441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d6ae5c4d4646ac94c2d4c3a4330ff1, State: Initialized, Role: FOLLOWER
I20260808 02:26:24.622421 18441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 32817 } }
I20260808 02:26:24.622483 18441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:24.622524 18441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:24.622571 18441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:24.623103 18441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 32817 } }
I20260808 02:26:24.623210 18441 leader_election.cc:302] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d6ae5c4d4646ac94c2d4c3a4330ff1; no voters:
I20260808 02:26:24.623404 18441 leader_election.cc:288] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:24.623553 18444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:24.623704 18441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:24.623744 18444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [term 1 LEADER]: Becoming Leader. State: Replica: 67d6ae5c4d4646ac94c2d4c3a4330ff1, State: Running, Role: LEADER
I20260808 02:26:24.623880 18444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 32817 } }
I20260808 02:26:24.624284 18446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d6ae5c4d4646ac94c2d4c3a4330ff1. Latest consensus state: current_term: 1 leader_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 32817 } } }
I20260808 02:26:24.624369 18446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:24.624374 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6ae5c4d4646ac94c2d4c3a4330ff1" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 32817 } } }
I20260808 02:26:24.624599 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:24.624595 18453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:24.625242 18453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:24.625385 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18371
I20260808 02:26:24.625504 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:24.625695 31360 external_mini_cluster.cc:1428] Setting key 4b14cb54390769731a1f88668578f184
I20260808 02:26:24.626845 18453 catalog_manager.cc:1347] Generated new cluster ID: 0bbdb756d73142d5bc22872dc0c63fc3
I20260808 02:26:24.626906 18453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:24.632128 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.626136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.30.160.62:32817)
0808 02:26:24.626625 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.626633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.626664 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:26:24.626774 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.626781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.626866 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.626998 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.627359 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.628704 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.629824 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.630417 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.630524 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.631252 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.631256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.631256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:24.631448 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.631487 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:24.631500 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.631561 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.631659 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.631729 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":69,"threads_started":1}
I20260808 02:26:24.636986 18453 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:24.637437 18453 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:24.642777 18453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1: Generated new TSK 0
I20260808 02:26:24.642933 18453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:24.648492 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:32817
--builtin_ntp_servers=127.30.160.20:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:24.731130 18465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:24.731345 18465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:24.731381 18465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:24.732573 18465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:24.732635 18465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:24.732676 18465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:24.734020 18465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:24.734285 18465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:24.734583 18465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:24.740159 18471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.741950 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.742293 18470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.742480 18465 server_base.cc:1034] running on GCE node
I20260808 02:26:24.742899 18465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.743193 18465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.744362 18465 hybrid_clock.cc:648] HybridClock initialized: now 1786155984744331 us; error 47 us; skew 500 ppm
I20260808 02:26:24.745966 18465 webserver.cc:460] Webserver started at http://127.30.160.1:34673/ using document root <none> and password file <none>
I20260808 02:26:24.746233 18465 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.746290 18465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.746416 18465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.747825 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "b2f860e8572e44a2a02992ba91d944b5"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "561b3ba51656050e55acb1f1b5b1658c"
server_key_iv: "7cb9fb692d53b2dd8a3cdb96482eb4b4"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.748426 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "b2f860e8572e44a2a02992ba91d944b5"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "561b3ba51656050e55acb1f1b5b1658c"
server_key_iv: "7cb9fb692d53b2dd8a3cdb96482eb4b4"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.750253 18465 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:24.751151 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.751368 18465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:24.751462 18465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "b2f860e8572e44a2a02992ba91d944b5"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "561b3ba51656050e55acb1f1b5b1658c"
server_key_iv: "7cb9fb692d53b2dd8a3cdb96482eb4b4"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.751528 18465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:24.765836 18465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:24.766062 18465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:24.766165 18465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:24.766346 18465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:24.766593 18465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:24.766623 18465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.766642 18465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:24.766655 18465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.772359 18465 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:39369
I20260808 02:26:24.772439 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:39369 every 8 connection(s)
I20260808 02:26:24.772748 18465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:24.774830 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18465
I20260808 02:26:24.774928 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:24.775146 31360 external_mini_cluster.cc:1428] Setting key 7c31118f3c7c2f247f869bdb9f9b4fa6
I20260808 02:26:24.776156 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:32817
--builtin_ntp_servers=127.30.160.20:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:26:24.778687 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.773419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:49405 (local address 127.30.160.62:32817)
0808 02:26:24.773469 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.773471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.773772 (+ 301us) server_negotiation.cc:407] Connection header received
0808 02:26:24.774215 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.774216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.774226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.774279 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.775006 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.775666 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.777187 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.777381 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.777434 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.778368 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.778369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.778369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:24.778374 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.778396 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:24.778400 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.778480 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.778585 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.778610 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:24.778932 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.773439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:32817 (local address 127.30.160.1:49405)
0808 02:26:24.773645 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.773676 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.774138 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.774346 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.774349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.774357 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:24.774863 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.774867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.775798 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.775798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.777086 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.777089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.777464 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.777465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.777619 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.778222 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:24.778222 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:24.778249 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:24.778270 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:24.778480 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:24.778487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:24.778490 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.778570 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.778665 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260808 02:26:24.779388 18599 heartbeater.cc:347] Connected to a master server at 127.30.160.62:32817
I20260808 02:26:24.779488 18599 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:24.779659 18599 heartbeater.cc:511] Master 127.30.160.62:32817 requested a full tablet report, sending...
I20260808 02:26:24.780195 18402 ts_manager.cc:194] Registered new tserver with Master: b2f860e8572e44a2a02992ba91d944b5 (127.30.160.1:39369)
I20260808 02:26:24.781150 18402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.1:49405
W20260808 02:26:24.858223 18602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:24.858410 18602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:24.858434 18602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:24.859508 18602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:24.859539 18602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:24.859562 18602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:24.860807 18602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:24.860996 18602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:24.861217 18602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:24.865996 18607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.866127 18602 server_base.cc:1034] running on GCE node
W20260808 02:26:24.866026 18608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.865996 18610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.866473 18602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.866725 18602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.867928 18602 hybrid_clock.cc:648] HybridClock initialized: now 1786155984867883 us; error 35 us; skew 500 ppm
I20260808 02:26:24.869156 18602 webserver.cc:460] Webserver started at http://127.30.160.2:44679/ using document root <none> and password file <none>
I20260808 02:26:24.869400 18602 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.869470 18602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.869608 18602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.870610 18602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "38d4d4a79188497baf2466e3c28d012f"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "839f8fd973e007a1fa5ee5a76a5accfb"
server_key_iv: "438e159484bcb5be1ec3b06eeccdde27"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.870976 18602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "38d4d4a79188497baf2466e3c28d012f"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "839f8fd973e007a1fa5ee5a76a5accfb"
server_key_iv: "438e159484bcb5be1ec3b06eeccdde27"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.872334 18602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:24.873072 18617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.873270 18602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:26:24.873371 18602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "38d4d4a79188497baf2466e3c28d012f"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "839f8fd973e007a1fa5ee5a76a5accfb"
server_key_iv: "438e159484bcb5be1ec3b06eeccdde27"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.873445 18602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:24.884791 18602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:24.885052 18602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:24.885195 18602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:24.885409 18602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:24.885707 18602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:24.885761 18602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.885795 18602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:24.885821 18602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.891397 18602 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:37211
I20260808 02:26:24.891429 18736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:37211 every 8 connection(s)
I20260808 02:26:24.891726 18602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:24.892128 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18602
I20260808 02:26:24.892262 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:24.892462 31360 external_mini_cluster.cc:1428] Setting key a9b5a5f359ca2d8bd074cf8d4070e6d1
I20260808 02:26:24.893422 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:32817
--builtin_ntp_servers=127.30.160.20:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:26:24.897531 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.892463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:59341 (local address 127.30.160.62:32817)
0808 02:26:24.892516 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:24.892519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:24.892798 (+ 279us) server_negotiation.cc:407] Connection header received
0808 02:26:24.893349 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:24.893350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:24.893361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:24.893401 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:24.894351 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.894822 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.896009 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.896249 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.896331 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.897250 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:24.897251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:24.897252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:24.897257 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:24.897272 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:24.897277 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:24.897339 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:24.897420 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 02:26:24.897453 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:26:24.897820 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:24.892457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:32817 (local address 127.30.160.2:59341)
0808 02:26:24.892673 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:24.892733 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:24.893243 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:24.893474 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:24.893479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:24.893489 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:24.894263 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.894270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.894907 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.894909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.895935 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:24.895939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:24.896339 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:24.896340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:24.896495 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:24.897063 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:24.897064 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:24.897111 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:24.897137 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:24.897354 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:24.897360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:24.897363 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:24.897424 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:26:24.897520 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260808 02:26:24.899928 18737 heartbeater.cc:347] Connected to a master server at 127.30.160.62:32817
I20260808 02:26:24.900040 18737 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:24.900240 18737 heartbeater.cc:511] Master 127.30.160.62:32817 requested a full tablet report, sending...
I20260808 02:26:24.900630 18402 ts_manager.cc:194] Registered new tserver with Master: 38d4d4a79188497baf2466e3c28d012f (127.30.160.2:37211)
I20260808 02:26:24.901154 18402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.2:59341
W20260808 02:26:24.980131 18740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:24.980420 18740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:24.980458 18740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:24.982048 18740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:24.982116 18740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:24.982159 18740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:24.983726 18740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:32817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:24.984015 18740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:24.984315 18740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:24.990195 18746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.990203 18748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:24.990394 18745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:24.990404 18740 server_base.cc:1034] running on GCE node
I20260808 02:26:24.990660 18740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:24.990971 18740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:24.992146 18740 hybrid_clock.cc:648] HybridClock initialized: now 1786155984992129 us; error 26 us; skew 500 ppm
I20260808 02:26:24.993882 18740 webserver.cc:460] Webserver started at http://127.30.160.3:33405/ using document root <none> and password file <none>
I20260808 02:26:24.994184 18740 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:24.994256 18740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:24.994405 18740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:24.995882 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "060374acc8644c918cd91a5e40227f52"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "6ab8a1ced174890419fa2406c2e390ae"
server_key_iv: "6a5966c98e9bc5af82e27faa7444cfa6"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.996397 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "060374acc8644c918cd91a5e40227f52"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "6ab8a1ced174890419fa2406c2e390ae"
server_key_iv: "6a5966c98e9bc5af82e27faa7444cfa6"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.998086 18740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:24.999043 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:24.999215 18740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:24.999296 18740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "060374acc8644c918cd91a5e40227f52"
format_stamp: "Formatted at 2026-08-08 02:26:24 on dist-test-slave-zkpd"
server_key: "6ab8a1ced174890419fa2406c2e390ae"
server_key_iv: "6a5966c98e9bc5af82e27faa7444cfa6"
server_key_version: "encryptionkey@0"
I20260808 02:26:24.999363 18740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:25.033107 18740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:25.033442 18740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:25.033591 18740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:25.033862 18740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:25.034211 18740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:25.034267 18740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:25.034312 18740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:25.034337 18740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:25.040164 18740 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:35737
I20260808 02:26:25.040208 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:35737 every 8 connection(s)
I20260808 02:26:25.040553 18740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:26:25.045624 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.041315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:47903 (local address 127.30.160.62:32817)
0808 02:26:25.041369 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.041370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.041620 (+ 250us) server_negotiation.cc:407] Connection header received
0808 02:26:25.042107 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.042108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.042117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.042180 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:25.042871 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.043346 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.044380 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.044563 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.044616 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.045381 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:25.045382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:25.045382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:25.045386 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:25.045400 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:25.045404 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:25.045460 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.045542 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.045565 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:25.045837 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.041274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:32817 (local address 127.30.160.3:47903)
0808 02:26:25.041490 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.041532 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:25.042022 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:25.042261 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:25.042265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:25.042274 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:25.042771 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.042775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.043426 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.043426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.044306 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.044309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.044641 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.044641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.044749 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.045253 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:25.045254 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:25.045281 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:25.045303 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:25.045472 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:25.045476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:25.045479 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:25.045530 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 02:26:25.045607 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260808 02:26:25.046378 18873 heartbeater.cc:347] Connected to a master server at 127.30.160.62:32817
I20260808 02:26:25.046468 18873 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:25.046607 18873 heartbeater.cc:511] Master 127.30.160.62:32817 requested a full tablet report, sending...
I20260808 02:26:25.046957 18402 ts_manager.cc:194] Registered new tserver with Master: 060374acc8644c918cd91a5e40227f52 (127.30.160.3:35737)
I20260808 02:26:25.047326 18402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.3:47903
I20260808 02:26:25.049682 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18740
I20260808 02:26:25.049808 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:25.049988 31360 external_mini_cluster.cc:1428] Setting key 40928be4fb5ea32e33d00e2ce8c9ba84
I20260808 02:26:25.051002 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:25.055142 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.052355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.30.160.62:32817)
0808 02:26:25.052451 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.052452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.052491 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:26:25.052538 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.052539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.052547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.052591 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:25.052750 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.053193 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.053925 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.054143 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.054195 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.054800 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:25.054801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:25.054801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:25.054805 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:25.054817 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:25.054820 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:25.054892 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.055019 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.055032 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:26:25.056690 18402 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:25.058096 18402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:25.064319 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.060290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.30.160.3:35737)
0808 02:26:25.060611 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.060616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.060627 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:25.060699 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.060701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.060718 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.060800 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:25.060978 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.061803 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.063175 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.063930 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.063999 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.064081 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.064201 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.064240 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":96,"threads_started":1}
I20260808 02:26:25.064319 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.060220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:35737 (local address 127.0.0.1:58756)
0808 02:26:25.060561 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.060573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:25.060629 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:25.060812 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:25.060815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:25.060817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:25.060878 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.060880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.061915 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.061916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.063078 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.063080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.064002 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.064003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.064089 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.064107 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:26:25.064179 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:26:25.064231 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260808 02:26:25.064384 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.060220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39369 (local address 127.0.0.1:51948)
0808 02:26:25.060391 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.060443 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:25.060503 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:25.060920 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:25.060921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:25.060923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:25.060969 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.060971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.062342 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.062343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.063362 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.063364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.064151 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.064152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.064211 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.064214 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:25.064243 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 02:26:25.064272 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260808 02:26:25.064815 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.060418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:37211 (local address 127.0.0.1:55878)
0808 02:26:25.060648 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.060658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:25.060705 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:25.061218 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:25.061219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:25.061221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:25.061266 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.061268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.062551 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.062552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.063481 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.063484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.064560 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.064561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.064633 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.064637 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:25.064691 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 02:26:25.064729 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260808 02:26:25.064965 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.060405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.30.160.1:39369)
0808 02:26:25.060763 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.060767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.060777 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:25.060825 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.060827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.060842 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.060908 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:25.061104 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.062242 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.063445 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.064054 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.064683 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.064783 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.064818 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.064846 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":83,"threads_started":1}
I20260808 02:26:25.065089 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.060625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.30.160.2:37211)
0808 02:26:25.060821 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.060825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.060835 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:25.060871 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.060873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.061034 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.061107 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:25.061352 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.062461 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.063575 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.064363 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.064437 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.064519 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.064971 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.065008 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260808 02:26:25.065187 18805 tablet_service.cc:1467] Processing CreateTablet for tablet 9b25ed11641c483096fe2c0d35a50caf (DEFAULT_TABLE table=test-table [id=8f33373976464302a004126154c70d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:25.065542 18805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b25ed11641c483096fe2c0d35a50caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:25.065611 18530 tablet_service.cc:1467] Processing CreateTablet for tablet 9b25ed11641c483096fe2c0d35a50caf (DEFAULT_TABLE table=test-table [id=8f33373976464302a004126154c70d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:25.065711 18667 tablet_service.cc:1467] Processing CreateTablet for tablet 9b25ed11641c483096fe2c0d35a50caf (DEFAULT_TABLE table=test-table [id=8f33373976464302a004126154c70d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:25.065986 18530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b25ed11641c483096fe2c0d35a50caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:25.066021 18667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b25ed11641c483096fe2c0d35a50caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:25.068226 18887 tablet_bootstrap.cc:492] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Bootstrap starting.
I20260808 02:26:25.068840 18887 tablet_bootstrap.cc:654] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:25.069152 18887 log.cc:824] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:25.069409 18888 tablet_bootstrap.cc:492] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5: Bootstrap starting.
I20260808 02:26:25.069720 18889 tablet_bootstrap.cc:492] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Bootstrap starting.
I20260808 02:26:25.070057 18887 tablet_bootstrap.cc:492] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: No bootstrap required, opened a new log
I20260808 02:26:25.070147 18887 ts_tablet_manager.cc:1397] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:25.070255 18888 tablet_bootstrap.cc:654] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:25.070379 18889 tablet_bootstrap.cc:654] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:25.070596 18888 log.cc:824] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:25.070689 18889 log.cc:824] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:25.071401 18889 tablet_bootstrap.cc:492] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: No bootstrap required, opened a new log
I20260808 02:26:25.071401 18888 tablet_bootstrap.cc:492] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5: No bootstrap required, opened a new log
I20260808 02:26:25.071470 18888 ts_tablet_manager.cc:1397] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:25.071470 18889 ts_tablet_manager.cc:1397] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:25.072644 18887 raft_consensus.cc:348] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.072798 18887 raft_consensus.cc:374] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:25.072839 18887 raft_consensus.cc:729] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060374acc8644c918cd91a5e40227f52, State: Initialized, Role: FOLLOWER
I20260808 02:26:25.072955 18887 consensus_queue.cc:260] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.073215 18887 ts_tablet_manager.cc:1428] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:26:25.073344 18873 heartbeater.cc:503] Master 127.30.160.62:32817 was elected leader, sending a full tablet report...
I20260808 02:26:25.074134 18888 raft_consensus.cc:348] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.074134 18889 raft_consensus.cc:348] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.074291 18889 raft_consensus.cc:374] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:25.074292 18888 raft_consensus.cc:374] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:25.074329 18888 raft_consensus.cc:729] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f860e8572e44a2a02992ba91d944b5, State: Initialized, Role: FOLLOWER
I20260808 02:26:25.074329 18889 raft_consensus.cc:729] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d4d4a79188497baf2466e3c28d012f, State: Initialized, Role: FOLLOWER
I20260808 02:26:25.074440 18889 consensus_queue.cc:260] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.074440 18888 consensus_queue.cc:260] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.074689 18888 ts_tablet_manager.cc:1428] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:26:25.074689 18889 ts_tablet_manager.cc:1428] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:26:25.074764 18737 heartbeater.cc:503] Master 127.30.160.62:32817 was elected leader, sending a full tablet report...
I20260808 02:26:25.075021 18599 heartbeater.cc:503] Master 127.30.160.62:32817 was elected leader, sending a full tablet report...
I20260808 02:26:25.138499 18893 raft_consensus.cc:482] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:25.138634 18893 raft_consensus.cc:504] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.139109 18893 leader_election.cc:288] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2f860e8572e44a2a02992ba91d944b5 (127.30.160.1:39369), 38d4d4a79188497baf2466e3c28d012f (127.30.160.2:37211)
I20260808 02:26:25.141961 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.139299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39369 (local address 127.30.160.3:44431)
0808 02:26:25.139380 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.139391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:25.139437 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:25.139594 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:25.139595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:25.139597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:25.139669 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.139670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.140384 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.140385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.141144 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.141146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.141674 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.141675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.141751 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.141754 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:25.141835 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:26:25.141886 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":1}
I20260808 02:26:25.141961 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.139369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:44431 (local address 127.30.160.1:39369)
0808 02:26:25.139424 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.139426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.139437 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:25.139496 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.139496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.139507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.139576 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:25.139743 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.140301 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.141245 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.141594 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.141677 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.141686 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.141855 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.141895 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:26:25.142068 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.139383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:48367 (local address 127.30.160.2:37211)
0808 02:26:25.139431 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.139433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.139568 (+ 135us) server_negotiation.cc:407] Connection header received
0808 02:26:25.139640 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.139641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.139654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.139739 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:25.139928 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.140384 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.141324 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.141672 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.141774 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.141800 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.141968 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.142008 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:26:25.142077 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.139322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:37211 (local address 127.30.160.3:48367)
0808 02:26:25.139490 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.139505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:25.139583 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:25.139745 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:25.139746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:25.139748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:25.139819 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.139825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.140470 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.140471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.141227 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:25.141229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.141794 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.141795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:25.141880 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.141883 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:25.141949 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 02:26:25.141988 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
W20260808 02:26:25.142336 18738 tablet.cc:2369] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:25.142375 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b25ed11641c483096fe2c0d35a50caf" candidate_uuid: "060374acc8644c918cd91a5e40227f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f860e8572e44a2a02992ba91d944b5" is_pre_election: true
I20260808 02:26:25.142529 18550 raft_consensus.cc:2454] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060374acc8644c918cd91a5e40227f52 in term 0.
I20260808 02:26:25.142867 18758 leader_election.cc:302] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060374acc8644c918cd91a5e40227f52, b2f860e8572e44a2a02992ba91d944b5; no voters:
I20260808 02:26:25.142869 18687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b25ed11641c483096fe2c0d35a50caf" candidate_uuid: "060374acc8644c918cd91a5e40227f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d4d4a79188497baf2466e3c28d012f" is_pre_election: true
I20260808 02:26:25.143005 18687 raft_consensus.cc:2454] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060374acc8644c918cd91a5e40227f52 in term 0.
I20260808 02:26:25.143018 18893 raft_consensus.cc:2781] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:25.143085 18893 raft_consensus.cc:482] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:25.143110 18893 raft_consensus.cc:3037] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:25.143873 18893 raft_consensus.cc:504] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.144075 18893 leader_election.cc:288] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [CANDIDATE]: Term 1 election: Requested vote from peers b2f860e8572e44a2a02992ba91d944b5 (127.30.160.1:39369), 38d4d4a79188497baf2466e3c28d012f (127.30.160.2:37211)
I20260808 02:26:25.144219 18687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b25ed11641c483096fe2c0d35a50caf" candidate_uuid: "060374acc8644c918cd91a5e40227f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d4d4a79188497baf2466e3c28d012f"
I20260808 02:26:25.144243 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b25ed11641c483096fe2c0d35a50caf" candidate_uuid: "060374acc8644c918cd91a5e40227f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f860e8572e44a2a02992ba91d944b5"
I20260808 02:26:25.144311 18687 raft_consensus.cc:3037] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:25.144340 18550 raft_consensus.cc:3037] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:25.145057 18687 raft_consensus.cc:2454] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060374acc8644c918cd91a5e40227f52 in term 1.
I20260808 02:26:25.145144 18550 raft_consensus.cc:2454] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060374acc8644c918cd91a5e40227f52 in term 1.
I20260808 02:26:25.145229 18759 leader_election.cc:302] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060374acc8644c918cd91a5e40227f52, 38d4d4a79188497baf2466e3c28d012f; no voters:
I20260808 02:26:25.145359 18893 raft_consensus.cc:2781] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:25.145489 18893 raft_consensus.cc:686] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 1 LEADER]: Becoming Leader. State: Replica: 060374acc8644c918cd91a5e40227f52, State: Running, Role: LEADER
I20260808 02:26:25.145560 18893 consensus_queue.cc:237] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } }
I20260808 02:26:25.146353 18401 catalog_manager.cc:5495] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 060374acc8644c918cd91a5e40227f52 (127.30.160.3). New cstate: current_term: 1 leader_uuid: "060374acc8644c918cd91a5e40227f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060374acc8644c918cd91a5e40227f52" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 } health_report { overall_health: UNKNOWN } } }
I20260808 02:26:25.160022 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.157691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.30.160.3:35737)
0808 02:26:25.157750 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:25.157752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:25.157759 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:26:25.157787 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:25.157788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:25.157805 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:25.157862 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:25.157957 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.158387 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.159078 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:25.159309 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:25.159364 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:25.159555 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:25.159688 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:25.159805 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:25.159865 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 02:26:25.159873 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:26:25.161761 18550 raft_consensus.cc:1261] T 9b25ed11641c483096fe2c0d35a50caf P b2f860e8572e44a2a02992ba91d944b5 [term 1 FOLLOWER]: Refusing update from remote peer 060374acc8644c918cd91a5e40227f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:25.161796 18687 raft_consensus.cc:1261] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 1 FOLLOWER]: Refusing update from remote peer 060374acc8644c918cd91a5e40227f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:25.162124 18893 consensus_queue.cc:1035] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f860e8572e44a2a02992ba91d944b5" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:25.162272 18897 consensus_queue.cc:1035] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d4d4a79188497baf2466e3c28d012f" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:25.165184 18901 mvcc.cc:205] Tried to move back new op lower bound from 7316094915220369408 to 7316094915156877312. Current Snapshot: MvccSnapshot[applied={T|T < 7316094915220369408}]
I20260808 02:26:25.166517 18401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:25.166631 18401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:25.167379 18401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d6ae5c4d4646ac94c2d4c3a4330ff1: Sending DeleteTablet for 3 replicas of tablet 9b25ed11641c483096fe2c0d35a50caf
I20260808 02:26:25.167691 18530 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b25ed11641c483096fe2c0d35a50caf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:25 UTC) from {username='slave'} at 127.0.0.1:51948
I20260808 02:26:25.167768 18805 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b25ed11641c483096fe2c0d35a50caf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:25 UTC) from {username='slave'} at 127.0.0.1:58756
I20260808 02:26:25.167994 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18465
I20260808 02:26:25.168017 18667 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b25ed11641c483096fe2c0d35a50caf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:25 UTC) from {username='slave'} at 127.0.0.1:55878
I20260808 02:26:25.171651 18912 tablet_replica.cc:330] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: stopping tablet replica
I20260808 02:26:25.171738 18912 raft_consensus.cc:2229] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:25.171862 18912 raft_consensus.cc:2258] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:25.172288 18912 ts_tablet_manager.cc:1905] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:25.172421 18911 tablet_replica.cc:330] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: stopping tablet replica
I20260808 02:26:25.172521 18911 raft_consensus.cc:2229] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:26:25.172614 18911 raft_consensus.cc:2258] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:25.173091 18911 ts_tablet_manager.cc:1905] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:25.174116 18912 ts_tablet_manager.cc:1918] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:25.174198 18912 log.cc:1196] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/9b25ed11641c483096fe2c0d35a50caf
I20260808 02:26:25.174511 18912 ts_tablet_manager.cc:1939] T 9b25ed11641c483096fe2c0d35a50caf P 38d4d4a79188497baf2466e3c28d012f: Deleting consensus metadata
I20260808 02:26:25.174924 18389 catalog_manager.cc:4841] TS 38d4d4a79188497baf2466e3c28d012f (127.30.160.2:37211): tablet 9b25ed11641c483096fe2c0d35a50caf (table test-table [id=8f33373976464302a004126154c70d23]) successfully deleted
I20260808 02:26:25.175084 18911 ts_tablet_manager.cc:1918] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:25.175182 18911 log.cc:1196] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/9b25ed11641c483096fe2c0d35a50caf
I20260808 02:26:25.175420 18911 ts_tablet_manager.cc:1939] T 9b25ed11641c483096fe2c0d35a50caf P 060374acc8644c918cd91a5e40227f52: Deleting consensus metadata
I20260808 02:26:25.175805 18389 catalog_manager.cc:4841] TS 060374acc8644c918cd91a5e40227f52 (127.30.160.3:35737): tablet 9b25ed11641c483096fe2c0d35a50caf (table test-table [id=8f33373976464302a004126154c70d23]) successfully deleted
W20260808 02:26:25.177402 18388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:39369 (error 108)
I20260808 02:26:25.177628 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18602
I20260808 02:26:25.178723 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:25.178586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:39369 (local address 127.0.0.1:51960)
0808 02:26:25.178642 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:25.178682 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:39369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 02:26:25.178969 18388 catalog_manager.cc:4584] TS b2f860e8572e44a2a02992ba91d944b5 (127.30.160.1:39369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b25ed11641c483096fe2c0d35a50caf: Network error: Client connection negotiation failed: client connection to 127.30.160.1:39369: connect: Connection refused (error 111)
I20260808 02:26:25.182971 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18740
I20260808 02:26:25.188949 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18371
2026-08-08T02:26:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (726 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:25 dist-test-slave-zkpd krb5kdc[18916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:25 dist-test-slave-zkpd krb5kdc[18916](info): set up 2 sockets
Aug 08 02:26:25 dist-test-slave-zkpd krb5kdc[18916](info): commencing operation
krb5kdc: starting...
W20260808 02:26:26.247807 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:26.367090 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:35127
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:35127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:26.447860 18930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:26.448069 18930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:26.448091 18930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:26.449169 18930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:26.449201 18930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:26.449215 18930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:26.449227 18930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:26.449239 18930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:26.450572 18930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:35127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:26.450745 18930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:26.450899 18930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:26:26.455830 18930 server_base.cc:1034] running on GCE node
W20260808 02:26:26.455803 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:26.455821 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:26.456012 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:26.456277 18930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:26.456563 18930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:26.457702 18930 hybrid_clock.cc:648] HybridClock initialized: now 1786155986457681 us; error 37 us; skew 500 ppm
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:26.460206 18930 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:26.460638 18930 webserver.cc:460] Webserver started at http://127.30.160.62:38285/ using document root <none> and password file <none>
I20260808 02:26:26.460812 18930 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:26.460851 18930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:26.460947 18930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:26.461896 18930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "176aa292fc5c4a8791ace648501bf521"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "7f0d7632b9d242dc42987cb4aa381b82"
server_key_iv: "c538497b8110421aedcc879c42ea9bb3"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.462200 18930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "176aa292fc5c4a8791ace648501bf521"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "7f0d7632b9d242dc42987cb4aa381b82"
server_key_iv: "c538497b8110421aedcc879c42ea9bb3"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.463450 18930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:26.464138 18946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.464309 18930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.464411 18930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "176aa292fc5c4a8791ace648501bf521"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "7f0d7632b9d242dc42987cb4aa381b82"
server_key_iv: "c538497b8110421aedcc879c42ea9bb3"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.464490 18930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:26.479400 18930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:26.481848 18930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:26.482050 18930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:26.486274 18930 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:35127
I20260808 02:26:26.486321 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:35127 every 8 connection(s)
I20260808 02:26:26.486691 18930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:26.487599 19000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:26.490021 19000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521: Bootstrap starting.
I20260808 02:26:26.490561 19000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:26.490804 19000 log.cc:824] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:26.491492 19000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521: No bootstrap required, opened a new log
I20260808 02:26:26.491657 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 18930
I20260808 02:26:26.491779 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:26.492023 31360 external_mini_cluster.cc:1428] Setting key 55275c1893f868f668b2569e801231a8
I20260808 02:26:26.493650 19000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176aa292fc5c4a8791ace648501bf521" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 35127 } }
I20260808 02:26:26.493836 19000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:26.493876 19000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176aa292fc5c4a8791ace648501bf521, State: Initialized, Role: FOLLOWER
I20260808 02:26:26.494014 19000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176aa292fc5c4a8791ace648501bf521" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 35127 } }
I20260808 02:26:26.494103 19000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:26.494141 19000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:26.494194 19000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:26.495222 19000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176aa292fc5c4a8791ace648501bf521" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 35127 } }
I20260808 02:26:26.495314 19000 leader_election.cc:302] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 176aa292fc5c4a8791ace648501bf521; no voters:
I20260808 02:26:26.495486 19000 leader_election.cc:288] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:26.495517 19005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:26.495635 19005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [term 1 LEADER]: Becoming Leader. State: Replica: 176aa292fc5c4a8791ace648501bf521, State: Running, Role: LEADER
I20260808 02:26:26.495718 19005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176aa292fc5c4a8791ace648501bf521" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 35127 } }
I20260808 02:26:26.495775 19000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:26.496094 19006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "176aa292fc5c4a8791ace648501bf521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176aa292fc5c4a8791ace648501bf521" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 35127 } } }
I20260808 02:26:26.496168 19006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:26.496485 19007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176aa292fc5c4a8791ace648501bf521. Latest consensus state: current_term: 1 leader_uuid: "176aa292fc5c4a8791ace648501bf521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176aa292fc5c4a8791ace648501bf521" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 35127 } } }
I20260808 02:26:26.496613 19007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:26.496762 19011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:26.497394 19011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:26.499083 19011 catalog_manager.cc:1347] Generated new cluster ID: 07e6d863753b45ecbc688bf40ef18bbb
I20260808 02:26:26.499137 19011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:26.505440 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.492537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.30.160.62:35127)
0808 02:26:26.492726 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:26.492735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:26.492768 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:26:26.493644 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:26.493652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:26.493716 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:26.493853 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:26.494673 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.495759 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.496934 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.497529 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.498050 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:26.500850 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:26.500854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:26.500854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:26.500868 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:26.502845 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:26.503276 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:26.503279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:26.503279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:26.503407 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:26.503630 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:26.503631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:26.503632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:26.503994 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:26.505021 (+ 1027us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:26.505096 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:26:26.505235 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260808 02:26:26.515563 19011 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:26.516206 19011 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:26.530207 19011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521: Generated new TSK 0
I20260808 02:26:26.530418 19011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
Entry for principal kudu/127.30.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab.
I20260808 02:26:26.560848 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:35127
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:26.643716 19026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:26.643929 19026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:26.643952 19026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:26.645136 19026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:26.645210 19026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:26.645269 19026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:26.646731 19026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.1.keytab
--principal=kudu/127.30.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:26.646987 19026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:26.647229 19026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:26.652125 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:26.652197 19026 server_base.cc:1034] running on GCE node
W20260808 02:26:26.652133 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:26.652133 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:26.652590 19026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:26.652856 19026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:26.654021 19026 hybrid_clock.cc:648] HybridClock initialized: now 1786155986654019 us; error 33 us; skew 500 ppm
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:26.656432 19026 init.cc:377] Logged in from keytab as kudu/127.30.160.1@KRBTEST.COM (short username kudu)
I20260808 02:26:26.656867 19026 webserver.cc:460] Webserver started at http://127.30.160.1:36171/ using document root <none> and password file <none>
I20260808 02:26:26.657099 19026 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:26.657166 19026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:26.657315 19026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:26.658329 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "54d6c47e56bd4c09886961943213d208"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "7c34e130b98e62324f4b25ddbad6f959"
server_key_iv: "c9592798771f487dd5551d1b119d5dcb"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.658682 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "54d6c47e56bd4c09886961943213d208"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "7c34e130b98e62324f4b25ddbad6f959"
server_key_iv: "c9592798771f487dd5551d1b119d5dcb"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.660018 19026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:26.660730 19041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.660909 19026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.661017 19026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "54d6c47e56bd4c09886961943213d208"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "7c34e130b98e62324f4b25ddbad6f959"
server_key_iv: "c9592798771f487dd5551d1b119d5dcb"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.661103 19026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:26.674140 19026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:26.676023 19026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:26.676179 19026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:26.676400 19026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:26.676704 19026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:26.676765 19026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.676811 19026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:26.676842 19026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.682410 19026 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:45039
I20260808 02:26:26.682513 19160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:45039 every 8 connection(s)
I20260808 02:26:26.682803 19026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:26.685738 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19026
I20260808 02:26:26.685885 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:26.686089 31360 external_mini_cluster.cc:1428] Setting key 561ecb1a93a4481865610ff790fcd373
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.1@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:26.692782 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.683562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:58783 (local address 127.30.160.62:35127)
0808 02:26:26.683630 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:26.683632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:26.683795 (+ 163us) server_negotiation.cc:407] Connection header received
0808 02:26:26.684262 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:26.684263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:26.684272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:26.684350 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:26.685240 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.685733 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.687073 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.687294 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.687356 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:26.689793 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:26.689794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:26.689795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:26.689803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:26.691268 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:26.691771 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:26.691772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:26.691772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:26.691824 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:26.692135 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:26.692135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:26.692136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:26.692239 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:26.692307 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:26.692637 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 02:26:26.692705 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:26:26.693096 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.683504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:35127 (local address 127.30.160.1:58783)
0808 02:26:26.683664 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:26.683703 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:26.684175 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:26.684413 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:26.684417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:26.684968 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:26.685158 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:26.685162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.685827 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.685829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:26.686981 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:26.686985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.687385 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.687386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:26.687534 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:26.688143 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:26.688153 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:26.689692 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:26.691370 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:26.691378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:26.691380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:26.691690 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:26.691909 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:26.691916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:26.691916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:26.692062 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:26.692308 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:26.692315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:26.692486 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 02:26:26.692604 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:26:26.692799 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260808 02:26:26.693681 19162 heartbeater.cc:347] Connected to a master server at 127.30.160.62:35127
I20260808 02:26:26.693801 19162 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:26.694005 19162 heartbeater.cc:511] Master 127.30.160.62:35127 requested a full tablet report, sending...
I20260808 02:26:26.694527 18963 ts_manager.cc:194] Registered new tserver with Master: 54d6c47e56bd4c09886961943213d208 (127.30.160.1:45039)
I20260808 02:26:26.695144 18963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.1@KRBTEST.COM'} at 127.30.160.1:58783
WARNING: no policy specified for kudu/127.30.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
Entry for principal kudu/127.30.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab.
I20260808 02:26:26.714092 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:35127
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:26.795289 19167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:26.795480 19167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:26.795502 19167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:26.796664 19167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:26.796701 19167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:26.796726 19167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:26.798161 19167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.2.keytab
--principal=kudu/127.30.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:26.798331 19167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:26.798496 19167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:26.803292 19173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:26.803335 19172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:26.803292 19175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:26.803625 19167 server_base.cc:1034] running on GCE node
I20260808 02:26:26.803838 19167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:26.804083 19167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:26.805222 19167 hybrid_clock.cc:648] HybridClock initialized: now 1786155986805208 us; error 48 us; skew 500 ppm
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:26.807612 19167 init.cc:377] Logged in from keytab as kudu/127.30.160.2@KRBTEST.COM (short username kudu)
I20260808 02:26:26.808100 19167 webserver.cc:460] Webserver started at http://127.30.160.2:36223/ using document root <none> and password file <none>
I20260808 02:26:26.808327 19167 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:26.808372 19167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:26.808466 19167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:26.809465 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "4fefe8941c0d42919b0159e7ea117c99"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "d609290979228c3044e2d9aee4920475"
server_key_iv: "e665a7d6815fc7cd327804dfd6d54494"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.809851 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "4fefe8941c0d42919b0159e7ea117c99"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "d609290979228c3044e2d9aee4920475"
server_key_iv: "e665a7d6815fc7cd327804dfd6d54494"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.811316 19167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:26.812093 19182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.812268 19167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:26.812368 19167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "4fefe8941c0d42919b0159e7ea117c99"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "d609290979228c3044e2d9aee4920475"
server_key_iv: "e665a7d6815fc7cd327804dfd6d54494"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.812450 19167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:26.825659 19167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:26.827402 19167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:26.827569 19167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:26.827785 19167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:26.828104 19167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:26.828159 19167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.828202 19167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:26.828228 19167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.834293 19167 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:44407
I20260808 02:26:26.834342 19298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:44407 every 8 connection(s)
I20260808 02:26:26.834636 19167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:26.838717 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19167
I20260808 02:26:26.838863 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:26.839068 31360 external_mini_cluster.cc:1428] Setting key fc2303235308a61a6ec8f384ceb82e5f
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.2@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:26.844322 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.835281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:35127 (local address 127.30.160.2:42659)
0808 02:26:26.835466 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:26.835506 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:26.836062 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:26.836301 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:26.836304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:26.836849 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:26.837039 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:26.837044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.837686 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.837686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:26.838460 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:26.838462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.838793 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.838795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:26.838987 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:26.839551 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:26.839559 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:26.841237 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:26.843005 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:26.843012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:26.843014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:26.843215 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:26.843428 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:26.843428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:26.843428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:26.843522 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:26.843792 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:26.843797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:26.843953 (+ 156us) client_negotiation.cc:769] Sending connection context
0808 02:26:26.844056 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:26:26.844173 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260808 02:26:26.844350 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.835323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:42659 (local address 127.30.160.62:35127)
0808 02:26:26.835373 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:26.835375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:26.835593 (+ 218us) server_negotiation.cc:407] Connection header received
0808 02:26:26.836157 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:26.836158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:26.836167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:26.836223 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:26.837116 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.837609 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.838532 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.838713 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.838762 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:26.841340 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:26.841342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:26.841342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:26.841350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:26.842931 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:26.843296 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:26.843297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:26.843298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:26.843361 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:26.843585 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:26.843586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:26.843586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:26.843728 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:26.843797 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:26.844105 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 02:26:26.844201 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:26:26.844942 19299 heartbeater.cc:347] Connected to a master server at 127.30.160.62:35127
I20260808 02:26:26.845041 19299 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:26.845216 19299 heartbeater.cc:511] Master 127.30.160.62:35127 requested a full tablet report, sending...
I20260808 02:26:26.845583 18963 ts_manager.cc:194] Registered new tserver with Master: 4fefe8941c0d42919b0159e7ea117c99 (127.30.160.2:44407)
I20260808 02:26:26.846112 18963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.2@KRBTEST.COM'} at 127.30.160.2:42659
WARNING: no policy specified for kudu/127.30.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
Entry for principal kudu/127.30.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab.
I20260808 02:26:26.866035 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:35127
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:26.948932 19305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:26.949122 19305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:26.949146 19305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:26.950273 19305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:26.950325 19305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:26.950371 19305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:26.951761 19305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.3.keytab
--principal=kudu/127.30.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:26.951997 19305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:26.952189 19305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:26.956970 19310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:26.957036 19305 server_base.cc:1034] running on GCE node
W20260808 02:26:26.956949 19313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:26.956969 19311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:26.957408 19305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:26.957604 19305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:26.958747 19305 hybrid_clock.cc:648] HybridClock initialized: now 1786155986958736 us; error 28 us; skew 500 ppm
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:26.961246 19305 init.cc:377] Logged in from keytab as kudu/127.30.160.3@KRBTEST.COM (short username kudu)
I20260808 02:26:26.961714 19305 webserver.cc:460] Webserver started at http://127.30.160.3:41541/ using document root <none> and password file <none>
I20260808 02:26:26.961942 19305 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:26.962013 19305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:26.962174 19305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:26.963270 19305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "3d50943be21547adb226cbbe2fc7c0fb"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "247a6e4fdb6d5e01614de2ac23e233e4"
server_key_iv: "7b69fcd7457beeba0b5853d00d7449e1"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.963634 19305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "3d50943be21547adb226cbbe2fc7c0fb"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "247a6e4fdb6d5e01614de2ac23e233e4"
server_key_iv: "7b69fcd7457beeba0b5853d00d7449e1"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.965135 19305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260808 02:26:26.965855 19320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.966020 19305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:26.966112 19305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "3d50943be21547adb226cbbe2fc7c0fb"
format_stamp: "Formatted at 2026-08-08 02:26:26 on dist-test-slave-zkpd"
server_key: "247a6e4fdb6d5e01614de2ac23e233e4"
server_key_iv: "7b69fcd7457beeba0b5853d00d7449e1"
server_key_version: "encryptionkey@0"
I20260808 02:26:26.966166 19305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:26.985621 19305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:26.987373 19305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:26.987491 19305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:26.987668 19305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:26.988032 19305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:26.988070 19305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.988093 19305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:26.988142 19305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:26.994048 19305 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:42493
I20260808 02:26:26.994112 19438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:42493 every 8 connection(s)
I20260808 02:26:26.994418 19305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:26:26 dist-test-slave-zkpd krb5kdc[18916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.3@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:27.000993 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19305
I20260808 02:26:27.001123 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:27.001338 31360 external_mini_cluster.cc:1428] Setting key 0e504465f147742b4b67c88609c819ce
I20260808 02:26:27.004106 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.995126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:36045 (local address 127.30.160.62:35127)
0808 02:26:26.995167 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:26.995169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:26.995399 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:26:26.996017 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:26.996018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:26.996031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:26.996091 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:26.997103 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.997631 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.998619 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.998814 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.998880 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.000992 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.000994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.000995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:27.001003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.002684 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:27.003106 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:27.003107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:27.003107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:27.003173 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:27.003441 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:27.003442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:27.003442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:27.003559 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.003636 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.003930 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.004010 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:27.004317 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:26.995091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:35127 (local address 127.30.160.3:36045)
0808 02:26:26.995272 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:26.995308 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:26.995879 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:26.996165 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:26.996169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:26.996803 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:26.996999 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:26.997003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.997737 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.997738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:26.998522 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:26.998524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:26.998897 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:26.998897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:26.999001 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:26.999461 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:26.999469 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:27.000892 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:27.002780 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:27.002787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:27.002790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:27.003026 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:27.003242 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:27.003243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:27.003244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:27.003364 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:27.003626 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:27.003634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:27.003772 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.003882 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.004076 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260808 02:26:27.004815 19439 heartbeater.cc:347] Connected to a master server at 127.30.160.62:35127
I20260808 02:26:27.004916 19439 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:27.005057 19439 heartbeater.cc:511] Master 127.30.160.62:35127 requested a full tablet report, sending...
I20260808 02:26:27.005379 18963 ts_manager.cc:194] Registered new tserver with Master: 3d50943be21547adb226cbbe2fc7c0fb (127.30.160.3:42493)
I20260808 02:26:27.005743 18963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.160.3@KRBTEST.COM'} at 127.30.160.3:36045
I20260808 02:26:27.013070 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:27.020624 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.014322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.30.160.62:35127)
0808 02:26:27.014398 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.014400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.014513 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:26:27.014598 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.014599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.014609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.014665 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:27.015261 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.015732 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.016533 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.016735 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.016805 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.017988 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.017990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.017990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:27.017996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.019444 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:27.019802 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:27.019803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:27.019803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:27.019871 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:27.020092 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:27.020093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:27.020093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:27.020236 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.020343 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.020454 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.020523 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:26:27.022162 18963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:27.023641 18963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:27.029971 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.025797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:44407 (local address 127.0.0.1:50652)
0808 02:26:27.026033 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.026045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.026100 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.026383 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.026386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.026389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.026454 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.026456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.027829 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.027830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.028820 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.028823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.029661 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.029662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.029740 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.029753 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.029838 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.029893 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":22,"thread_start_us":53,"threads_started":1}
I20260808 02:26:27.030017 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.026025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.30.160.2:44407)
0808 02:26:27.026217 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.026221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.026230 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:27.026262 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.026263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.026275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.026369 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.026544 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.027715 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.028921 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.029505 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.029574 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.029627 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.029886 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.029930 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260808 02:26:27.030263 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.025809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:45039 (local address 127.0.0.1:41696)
0808 02:26:27.026168 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.026177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.026212 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.026609 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.026609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.026611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.026657 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.026659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.028045 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.028046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.029206 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.029209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.030082 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.030083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.030154 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.030158 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.030185 (+ 27us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.030216 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260808 02:26:27.030387 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.025797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:42493 (local address 127.0.0.1:40456)
0808 02:26:27.025967 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.026008 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.026075 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.026680 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.026681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.026683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.026719 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.026720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.027967 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.027968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.029206 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.029209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.030130 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.030132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.030192 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.030196 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.030225 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.030262 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":58,"thread_start_us":67,"threads_started":1}
I20260808 02:26:27.030550 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.025987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.30.160.3:42493)
0808 02:26:27.026497 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.026502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.026512 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:27.026552 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.026554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.026569 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.026676 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.026779 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.027813 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.029310 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.030022 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.030314 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.030387 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.030414 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.030456 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":79,"threads_started":1}
I20260808 02:26:27.030668 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.026190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.30.160.1:45039)
0808 02:26:27.026436 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.026441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.026452 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:27.026492 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.026494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.026510 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.026737 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.026828 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.027881 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.029316 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.029978 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.030357 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.030428 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.030534 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.030593 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260808 02:26:27.031242 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 67f0eed727d94b8596be17389517a4fc (DEFAULT_TABLE table=test-table [id=09e316eb78ce46ea8c312b90226e9348]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:27.031239 19094 tablet_service.cc:1467] Processing CreateTablet for tablet 67f0eed727d94b8596be17389517a4fc (DEFAULT_TABLE table=test-table [id=09e316eb78ce46ea8c312b90226e9348]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:27.031599 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f0eed727d94b8596be17389517a4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.031605 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f0eed727d94b8596be17389517a4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.031543 19233 tablet_service.cc:1467] Processing CreateTablet for tablet 67f0eed727d94b8596be17389517a4fc (DEFAULT_TABLE table=test-table [id=09e316eb78ce46ea8c312b90226e9348]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:27.031857 19233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f0eed727d94b8596be17389517a4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.034366 19454 tablet_bootstrap.cc:492] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Bootstrap starting.
I20260808 02:26:27.035140 19454 tablet_bootstrap.cc:654] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.035152 19455 tablet_bootstrap.cc:492] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: Bootstrap starting.
I20260808 02:26:27.035751 19454 log.cc:824] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.035825 19455 tablet_bootstrap.cc:654] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.035983 19456 tablet_bootstrap.cc:492] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Bootstrap starting.
I20260808 02:26:27.036408 19455 log.cc:824] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.036626 19454 tablet_bootstrap.cc:492] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: No bootstrap required, opened a new log
I20260808 02:26:27.036674 19456 tablet_bootstrap.cc:654] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.036701 19454 ts_tablet_manager.cc:1397] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:27.037222 19456 log.cc:824] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.037328 19455 tablet_bootstrap.cc:492] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: No bootstrap required, opened a new log
I20260808 02:26:27.037395 19455 ts_tablet_manager.cc:1397] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:26:27.037957 19456 tablet_bootstrap.cc:492] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: No bootstrap required, opened a new log
I20260808 02:26:27.038035 19456 ts_tablet_manager.cc:1397] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:26:27.038770 19454 raft_consensus.cc:348] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.038969 19454 raft_consensus.cc:374] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.039011 19454 raft_consensus.cc:729] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fefe8941c0d42919b0159e7ea117c99, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.039134 19454 consensus_queue.cc:260] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.039436 19454 ts_tablet_manager.cc:1428] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:26:27.039472 19299 heartbeater.cc:503] Master 127.30.160.62:35127 was elected leader, sending a full tablet report...
I20260808 02:26:27.039480 19455 raft_consensus.cc:348] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.039633 19455 raft_consensus.cc:374] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.039654 19455 raft_consensus.cc:729] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d6c47e56bd4c09886961943213d208, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.039669 19456 raft_consensus.cc:348] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.039783 19455 consensus_queue.cc:260] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.039856 19456 raft_consensus.cc:374] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.039921 19456 raft_consensus.cc:729] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d50943be21547adb226cbbe2fc7c0fb, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.040030 19456 consensus_queue.cc:260] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.040091 19455 ts_tablet_manager.cc:1428] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:26:27.040220 19162 heartbeater.cc:503] Master 127.30.160.62:35127 was elected leader, sending a full tablet report...
I20260808 02:26:27.040282 19456 ts_tablet_manager.cc:1428] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:26:27.040390 19439 heartbeater.cc:503] Master 127.30.160.62:35127 was elected leader, sending a full tablet report...
W20260808 02:26:27.085167 19301 tablet.cc:2369] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:26:27.183550 19163 tablet.cc:2369] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:26:27.245023 19440 tablet.cc:2369] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:27.247503 19461 raft_consensus.cc:482] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:27.247607 19461 raft_consensus.cc:504] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.248008 19461 leader_election.cc:288] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fefe8941c0d42919b0159e7ea117c99 (127.30.160.2:44407), 3d50943be21547adb226cbbe2fc7c0fb (127.30.160.3:42493)
I20260808 02:26:27.252048 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.248270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:38793 (local address 127.30.160.2:44407)
0808 02:26:27.248315 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.248317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.248470 (+ 153us) server_negotiation.cc:407] Connection header received
0808 02:26:27.248508 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.248510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.248520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.248586 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.248813 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.249551 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.251085 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.251662 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.251742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.251751 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.251938 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.251977 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:26:27.252048 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.248326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:54051 (local address 127.30.160.3:42493)
0808 02:26:27.248371 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.248373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.248468 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:26:27.248508 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.248509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.248520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.248586 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.248794 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.249551 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.251085 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.251662 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.251742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.251751 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.251936 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.251981 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:26:27.252102 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.248213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:44407 (local address 127.30.160.1:38793)
0808 02:26:27.248368 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.248380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.248456 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.248597 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.248598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.248600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.248713 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.248716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.249636 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.249638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.250990 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.250994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.251751 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.251753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.251836 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.251845 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.251927 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.251983 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":65,"thread_start_us":103,"threads_started":1}
I20260808 02:26:27.252301 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.248228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:42493 (local address 127.30.160.1:54051)
0808 02:26:27.248369 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.248380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.248459 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.248597 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.248598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.248602 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.248723 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.248724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.249637 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.249638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.250991 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.250994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.251753 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.251754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.251837 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.251843 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.251931 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.251983 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":83,"thread_start_us":69,"threads_started":1}
I20260808 02:26:27.252507 19253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f0eed727d94b8596be17389517a4fc" candidate_uuid: "54d6c47e56bd4c09886961943213d208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fefe8941c0d42919b0159e7ea117c99" is_pre_election: true
I20260808 02:26:27.252686 19253 raft_consensus.cc:2454] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54d6c47e56bd4c09886961943213d208 in term 0.
I20260808 02:26:27.252781 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f0eed727d94b8596be17389517a4fc" candidate_uuid: "54d6c47e56bd4c09886961943213d208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d50943be21547adb226cbbe2fc7c0fb" is_pre_election: true
I20260808 02:26:27.252919 19045 leader_election.cc:302] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fefe8941c0d42919b0159e7ea117c99, 54d6c47e56bd4c09886961943213d208; no voters:
I20260808 02:26:27.252950 19391 raft_consensus.cc:2454] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54d6c47e56bd4c09886961943213d208 in term 0.
I20260808 02:26:27.253057 19461 raft_consensus.cc:2781] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:27.253146 19461 raft_consensus.cc:482] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:27.253185 19461 raft_consensus.cc:3037] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:27.253932 19461 raft_consensus.cc:504] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.254158 19461 leader_election.cc:288] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [CANDIDATE]: Term 1 election: Requested vote from peers 4fefe8941c0d42919b0159e7ea117c99 (127.30.160.2:44407), 3d50943be21547adb226cbbe2fc7c0fb (127.30.160.3:42493)
I20260808 02:26:27.254236 19253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f0eed727d94b8596be17389517a4fc" candidate_uuid: "54d6c47e56bd4c09886961943213d208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fefe8941c0d42919b0159e7ea117c99"
I20260808 02:26:27.254323 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f0eed727d94b8596be17389517a4fc" candidate_uuid: "54d6c47e56bd4c09886961943213d208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d50943be21547adb226cbbe2fc7c0fb"
I20260808 02:26:27.254328 19253 raft_consensus.cc:3037] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:27.254405 19391 raft_consensus.cc:3037] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:27.255059 19253 raft_consensus.cc:2454] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54d6c47e56bd4c09886961943213d208 in term 1.
I20260808 02:26:27.255191 19391 raft_consensus.cc:2454] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54d6c47e56bd4c09886961943213d208 in term 1.
I20260808 02:26:27.255223 19045 leader_election.cc:302] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fefe8941c0d42919b0159e7ea117c99, 54d6c47e56bd4c09886961943213d208; no voters:
I20260808 02:26:27.255344 19461 raft_consensus.cc:2781] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:27.255540 19461 raft_consensus.cc:686] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [term 1 LEADER]: Becoming Leader. State: Replica: 54d6c47e56bd4c09886961943213d208, State: Running, Role: LEADER
I20260808 02:26:27.255743 19461 consensus_queue.cc:237] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } }
I20260808 02:26:27.256582 18962 catalog_manager.cc:5495] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54d6c47e56bd4c09886961943213d208 (127.30.160.1). New cstate: current_term: 1 leader_uuid: "54d6c47e56bd4c09886961943213d208" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54d6c47e56bd4c09886961943213d208" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 45039 } health_report { overall_health: HEALTHY } } }
I20260808 02:26:27.315127 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.312711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.30.160.1:45039)
0808 02:26:27.312768 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.312770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.312779 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:27.312825 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.312826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.312844 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.312907 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:27.313044 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.313556 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.314403 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.314588 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.314673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.314852 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:27.314957 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:27.315018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.315051 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.315069 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:26:27.317306 19391 raft_consensus.cc:1261] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 1 FOLLOWER]: Refusing update from remote peer 54d6c47e56bd4c09886961943213d208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:27.317315 19253 raft_consensus.cc:1261] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 1 FOLLOWER]: Refusing update from remote peer 54d6c47e56bd4c09886961943213d208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:27.317734 19461 consensus_queue.cc:1035] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d50943be21547adb226cbbe2fc7c0fb" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 42493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:27.317931 19466 consensus_queue.cc:1035] T 67f0eed727d94b8596be17389517a4fc P 54d6c47e56bd4c09886961943213d208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fefe8941c0d42919b0159e7ea117c99" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:27.320785 19471 mvcc.cc:205] Tried to move back new op lower bound from 7316094924048285696 to 7316094923800326144. Current Snapshot: MvccSnapshot[applied={T|T < 7316094924048285696}]
I20260808 02:26:27.321149 19469 mvcc.cc:205] Tried to move back new op lower bound from 7316094924048285696 to 7316094923800326144. Current Snapshot: MvccSnapshot[applied={T|T < 7316094924048285696 or (T in {7316094924048285696})}]
I20260808 02:26:27.322237 18962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:27.322329 18962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:27.323061 18962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 176aa292fc5c4a8791ace648501bf521: Sending DeleteTablet for 3 replicas of tablet 67f0eed727d94b8596be17389517a4fc
I20260808 02:26:27.323385 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f0eed727d94b8596be17389517a4fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:27 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:40456
I20260808 02:26:27.323446 19233 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f0eed727d94b8596be17389517a4fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:27 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:50652
I20260808 02:26:27.323625 19478 tablet_replica.cc:330] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: stopping tablet replica
I20260808 02:26:27.323779 19094 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f0eed727d94b8596be17389517a4fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:27 UTC) from {username='kudu', principal='kudu/127.30.160.62@KRBTEST.COM'} at 127.0.0.1:41696
I20260808 02:26:27.323808 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19026
I20260808 02:26:27.325090 19479 tablet_replica.cc:330] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: stopping tablet replica
I20260808 02:26:27.325186 19479 raft_consensus.cc:2229] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:27.325299 19479 raft_consensus.cc:2258] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:27.325489 19478 raft_consensus.cc:2229] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:27.325615 19478 raft_consensus.cc:2258] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:27.326335 19479 ts_tablet_manager.cc:1905] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:27.326457 19478 ts_tablet_manager.cc:1905] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:27.328248 19478 ts_tablet_manager.cc:1918] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:27.328248 19479 ts_tablet_manager.cc:1918] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:27.328331 19479 log.cc:1196] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/67f0eed727d94b8596be17389517a4fc
I20260808 02:26:27.328331 19478 log.cc:1196] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/67f0eed727d94b8596be17389517a4fc
I20260808 02:26:27.328604 19479 ts_tablet_manager.cc:1939] T 67f0eed727d94b8596be17389517a4fc P 4fefe8941c0d42919b0159e7ea117c99: Deleting consensus metadata
I20260808 02:26:27.328604 19478 ts_tablet_manager.cc:1939] T 67f0eed727d94b8596be17389517a4fc P 3d50943be21547adb226cbbe2fc7c0fb: Deleting consensus metadata
I20260808 02:26:27.329016 18950 catalog_manager.cc:4841] TS 4fefe8941c0d42919b0159e7ea117c99 (127.30.160.2:44407): tablet 67f0eed727d94b8596be17389517a4fc (table test-table [id=09e316eb78ce46ea8c312b90226e9348]) successfully deleted
I20260808 02:26:27.329077 18947 catalog_manager.cc:4841] TS 3d50943be21547adb226cbbe2fc7c0fb (127.30.160.3:42493): tablet 67f0eed727d94b8596be17389517a4fc (table test-table [id=09e316eb78ce46ea8c312b90226e9348]) successfully deleted
W20260808 02:26:27.331347 18949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:45039 (error 108)
I20260808 02:26:27.331533 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19167
I20260808 02:26:27.331841 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.331684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:45039 (local address 127.0.0.1:41720)
0808 02:26:27.331725 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.331754 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 02:26:27.331970 18949 catalog_manager.cc:4584] TS 54d6c47e56bd4c09886961943213d208 (127.30.160.1:45039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67f0eed727d94b8596be17389517a4fc: Network error: Client connection negotiation failed: client connection to 127.30.160.1:45039: connect: Connection refused (error 111)
I20260808 02:26:27.338018 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19305
I20260808 02:26:27.345449 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 18930
2026-08-08T02:26:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2157 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T02:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:27Z Disabled control of system clock
I20260808 02:26:27.374742 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38391
--webserver_interface=127.30.160.62
--webserver_port=0
--builtin_ntp_servers=127.30.160.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:38391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:27.456077 19484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:27.456279 19484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:27.456307 19484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:27.457386 19484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:27.457418 19484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:27.457432 19484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:27.457443 19484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:27.457455 19484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:27.459151 19484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:38391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:27.459431 19484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:27.459841 19484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:27.466137 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.466162 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.466144 19489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:27.466377 19484 server_base.cc:1034] running on GCE node
I20260808 02:26:27.466602 19484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:27.466826 19484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:27.467963 19484 hybrid_clock.cc:648] HybridClock initialized: now 1786155987467945 us; error 32 us; skew 500 ppm
I20260808 02:26:27.469205 19484 webserver.cc:460] Webserver started at http://127.30.160.62:32837/ using document root <none> and password file <none>
I20260808 02:26:27.469475 19484 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:27.469521 19484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:27.469626 19484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:27.470597 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "a4aa6d31b7af471cb17e5e3b95a47767"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "8c6219a937222bc0fcdd08374aad908e"
server_key_iv: "7d4c22672f1d6d0e3211d375266a13c3"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.470885 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "a4aa6d31b7af471cb17e5e3b95a47767"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "8c6219a937222bc0fcdd08374aad908e"
server_key_iv: "7d4c22672f1d6d0e3211d375266a13c3"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.472115 19484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:27.472762 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.472944 19484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.473035 19484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "a4aa6d31b7af471cb17e5e3b95a47767"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "8c6219a937222bc0fcdd08374aad908e"
server_key_iv: "7d4c22672f1d6d0e3211d375266a13c3"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.473109 19484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:27.521566 19484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:27.521938 19484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:27.522109 19484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:27.526132 19484 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:38391
I20260808 02:26:27.526216 19555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:38391 every 8 connection(s)
I20260808 02:26:27.526564 19484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:27.527544 19556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.529860 19556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767: Bootstrap starting.
I20260808 02:26:27.530124 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19484
I20260808 02:26:27.530220 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:27.530427 31360 external_mini_cluster.cc:1428] Setting key a64833831d0801ead6f7221d6087baa4
I20260808 02:26:27.530553 19556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.530884 19556 log.cc:824] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.531662 19556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767: No bootstrap required, opened a new log
I20260808 02:26:27.534077 19556 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38391 } }
I20260808 02:26:27.534205 19556 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.534227 19556 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aa6d31b7af471cb17e5e3b95a47767, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.534354 19556 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38391 } }
I20260808 02:26:27.534440 19556 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:27.534463 19556 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:27.534531 19556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:27.535049 19556 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38391 } }
I20260808 02:26:27.535171 19556 leader_election.cc:302] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4aa6d31b7af471cb17e5e3b95a47767; no voters:
I20260808 02:26:27.535336 19556 leader_election.cc:288] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:27.535375 19562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:27.535545 19562 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [term 1 LEADER]: Becoming Leader. State: Replica: a4aa6d31b7af471cb17e5e3b95a47767, State: Running, Role: LEADER
I20260808 02:26:27.535610 19556 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:27.535647 19562 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38391 } }
I20260808 02:26:27.535982 19563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4aa6d31b7af471cb17e5e3b95a47767. Latest consensus state: current_term: 1 leader_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38391 } } }
I20260808 02:26:27.536073 19563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:27.536310 19569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:27.536427 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4aa6d31b7af471cb17e5e3b95a47767" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38391 } } }
I20260808 02:26:27.536576 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:27.536803 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.530954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.30.160.62:38391)
0808 02:26:27.531155 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.531164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.531194 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:26:27.531745 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.531755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.531836 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.532004 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:27.532233 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.533634 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.534559 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.535001 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.535137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.535911 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.535917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.535917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:27.536146 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.536187 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:27.536203 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.536270 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.536520 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.536605 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260808 02:26:27.536818 19569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:27.538687 19569 catalog_manager.cc:1347] Generated new cluster ID: 8a8d613eea0a42c4b1831cbce91ae7c8
I20260808 02:26:27.538745 19569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:27.546980 19569 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:27.547639 19569 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:27.559358 19569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767: Generated new TSK 0
I20260808 02:26:27.559544 19569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:27.561717 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.1:0
--local_ip_for_outbound_sockets=127.30.160.1
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38391
--builtin_ntp_servers=127.30.160.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:27.644764 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:27.644986 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:27.645020 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:27.646129 19583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:27.646189 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:27.646229 19583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.1
I20260808 02:26:27.647541 19583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.160.1
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.1
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:27.647789 19583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:27.648154 19583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:27.653743 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.653792 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.653753 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:27.653995 19583 server_base.cc:1034] running on GCE node
I20260808 02:26:27.654213 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:27.654445 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:27.655606 19583 hybrid_clock.cc:648] HybridClock initialized: now 1786155987655592 us; error 29 us; skew 500 ppm
I20260808 02:26:27.656906 19583 webserver.cc:460] Webserver started at http://127.30.160.1:42203/ using document root <none> and password file <none>
I20260808 02:26:27.657140 19583 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:27.657207 19583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:27.657364 19583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:27.658371 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data/instance:
uuid: "dc1d3d7980594f1b93af6c454704b177"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "d9b86a77c6791788a294c8446934741d"
server_key_iv: "bc27cd45e2aac4adc94665bcc8065bae"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.658727 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance:
uuid: "dc1d3d7980594f1b93af6c454704b177"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "d9b86a77c6791788a294c8446934741d"
server_key_iv: "bc27cd45e2aac4adc94665bcc8065bae"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.660125 19583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:27.660959 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.661134 19583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:27.661230 19583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal
uuid: "dc1d3d7980594f1b93af6c454704b177"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "d9b86a77c6791788a294c8446934741d"
server_key_iv: "bc27cd45e2aac4adc94665bcc8065bae"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.661310 19583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:27.673529 19583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:27.673837 19583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:27.673985 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:27.674201 19583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:27.674500 19583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:27.674553 19583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.674593 19583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:27.674621 19583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.680673 19583 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.1:38761
I20260808 02:26:27.680766 19718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.1:38761 every 8 connection(s)
I20260808 02:26:27.681084 19583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:27.686347 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.681794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:33791 (local address 127.30.160.62:38391)
0808 02:26:27.681841 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.681843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.682078 (+ 235us) server_negotiation.cc:407] Connection header received
0808 02:26:27.682600 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.682601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.682614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.682663 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:27.683373 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.683878 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.684860 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.685067 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.685141 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.686046 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.686047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.686047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:27.686052 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.686067 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:27.686071 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.686131 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.686240 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.686268 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:26:27.686566 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.681767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38391 (local address 127.30.160.1:33791)
0808 02:26:27.681939 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.681986 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.682494 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.682744 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.682748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.682763 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:27.683275 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.683280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.683992 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.683994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.684764 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.684766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.685162 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.685164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.685297 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.685892 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:27.685892 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:27.685925 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:27.685946 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:27.686148 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:27.686155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:27.686158 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.686240 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.686343 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260808 02:26:27.687054 19719 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38391
I20260808 02:26:27.687142 19719 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:27.687374 19719 heartbeater.cc:511] Master 127.30.160.62:38391 requested a full tablet report, sending...
I20260808 02:26:27.687362 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19583
I20260808 02:26:27.687480 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:27.687685 31360 external_mini_cluster.cc:1428] Setting key f392405dec533da288bee26e431e5e37
I20260808 02:26:27.687951 19519 ts_manager.cc:194] Registered new tserver with Master: dc1d3d7980594f1b93af6c454704b177 (127.30.160.1:38761)
I20260808 02:26:27.688738 19519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.1:33791
I20260808 02:26:27.688755 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.2:0
--local_ip_for_outbound_sockets=127.30.160.2
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38391
--builtin_ntp_servers=127.30.160.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:26:27.769893 19723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:27.770080 19723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:27.770102 19723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:27.771142 19723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:27.771173 19723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:27.771196 19723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.2
I20260808 02:26:27.772560 19723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.160.2
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.2
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:27.772779 19723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:27.773118 19723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:27.778029 19730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.777990 19729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.778004 19732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:27.778169 19723 server_base.cc:1034] running on GCE node
I20260808 02:26:27.778434 19723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:27.778694 19723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:27.779865 19723 hybrid_clock.cc:648] HybridClock initialized: now 1786155987779853 us; error 25 us; skew 500 ppm
I20260808 02:26:27.781118 19723 webserver.cc:460] Webserver started at http://127.30.160.2:45531/ using document root <none> and password file <none>
I20260808 02:26:27.781356 19723 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:27.781425 19723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:27.781577 19723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:27.782593 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data/instance:
uuid: "1f131811e9524d88bdfc011239f09b85"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "4ef4d5fdb0ed17846dfd7fc57852726a"
server_key_iv: "5f580ed85f58472cfc28f152b2bd4677"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.782963 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance:
uuid: "1f131811e9524d88bdfc011239f09b85"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "4ef4d5fdb0ed17846dfd7fc57852726a"
server_key_iv: "5f580ed85f58472cfc28f152b2bd4677"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.784313 19723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:27.785066 19738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.785243 19723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:27.785363 19723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal
uuid: "1f131811e9524d88bdfc011239f09b85"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "4ef4d5fdb0ed17846dfd7fc57852726a"
server_key_iv: "5f580ed85f58472cfc28f152b2bd4677"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.785437 19723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:27.795984 19723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:27.796231 19723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:27.796375 19723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:27.796583 19723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:27.796888 19723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:27.796948 19723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.796986 19723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:27.797013 19723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.802626 19723 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.2:38119
I20260808 02:26:27.802673 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.2:38119 every 8 connection(s)
I20260808 02:26:27.802958 19723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:27.804411 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19723
I20260808 02:26:27.804528 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:27.804741 31360 external_mini_cluster.cc:1428] Setting key 64deffd79ac73dae47d755ef52785840
I20260808 02:26:27.805696 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.160.3:0
--local_ip_for_outbound_sockets=127.30.160.3
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38391
--builtin_ntp_servers=127.30.160.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:26:27.808387 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.803617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.2:60459 (local address 127.30.160.62:38391)
0808 02:26:27.803661 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.803663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.803848 (+ 185us) server_negotiation.cc:407] Connection header received
0808 02:26:27.804354 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.804356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.804369 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.804459 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:27.805265 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.805864 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.806836 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.807159 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.807240 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.808133 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.808134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.808135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:27.808139 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.808153 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:27.808157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.808203 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.808305 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.808328 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:26:27.808665 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.803576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38391 (local address 127.30.160.2:60459)
0808 02:26:27.803743 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.803782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.804253 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.804537 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.804541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.804557 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:27.805180 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.805185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.805975 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.805977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.806748 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.806751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.807266 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.807267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.807443 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.807981 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:27.807982 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:27.808021 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:27.808049 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:27.808219 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:27.808223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:27.808226 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.808291 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.808400 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260808 02:26:27.809142 19857 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38391
I20260808 02:26:27.809237 19857 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:27.809422 19857 heartbeater.cc:511] Master 127.30.160.62:38391 requested a full tablet report, sending...
I20260808 02:26:27.809782 19519 ts_manager.cc:194] Registered new tserver with Master: 1f131811e9524d88bdfc011239f09b85 (127.30.160.2:38119)
I20260808 02:26:27.810154 19519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.2:60459
W20260808 02:26:27.888880 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:27.889065 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:27.889086 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:27.890215 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:27.890254 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:27.890276 19860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.160.3
I20260808 02:26:27.891619 19860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.160.3
--webserver_port=0
--tserver_master_addrs=127.30.160.62:38391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.160.3
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:27.891848 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:27.892154 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:27.896965 19870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:27.896972 19866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:27.897105 19860 server_base.cc:1034] running on GCE node
W20260808 02:26:27.896963 19868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:27.897365 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:27.897624 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:27.898809 19860 hybrid_clock.cc:648] HybridClock initialized: now 1786155987898804 us; error 34 us; skew 500 ppm
I20260808 02:26:27.900067 19860 webserver.cc:460] Webserver started at http://127.30.160.3:38111/ using document root <none> and password file <none>
I20260808 02:26:27.900326 19860 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:27.900395 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:27.900548 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:27.901587 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data/instance:
uuid: "6de14058ef0c46fa8bf357acfcf8ad6f"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "725b3687ceb5a2e76924787715bf7699"
server_key_iv: "92ffc67db7f1b9859bf6656261c830a3"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.901944 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance:
uuid: "6de14058ef0c46fa8bf357acfcf8ad6f"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "725b3687ceb5a2e76924787715bf7699"
server_key_iv: "92ffc67db7f1b9859bf6656261c830a3"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.903204 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:27.903878 19877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.904073 19860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:27.904170 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal
uuid: "6de14058ef0c46fa8bf357acfcf8ad6f"
format_stamp: "Formatted at 2026-08-08 02:26:27 on dist-test-slave-zkpd"
server_key: "725b3687ceb5a2e76924787715bf7699"
server_key_iv: "92ffc67db7f1b9859bf6656261c830a3"
server_key_version: "encryptionkey@0"
I20260808 02:26:27.904246 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:27.938695 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:27.939028 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:27.939188 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:27.939425 19860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:27.939775 19860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:27.939836 19860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.939878 19860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:27.939944 19860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:27.945884 19860 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.3:43199
I20260808 02:26:27.945933 19994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.3:43199 every 8 connection(s)
I20260808 02:26:27.946251 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/data/info.pb
I20260808 02:26:27.951376 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.946942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.3:32943 (local address 127.30.160.62:38391)
0808 02:26:27.947014 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.947016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.947276 (+ 260us) server_negotiation.cc:407] Connection header received
0808 02:26:27.947807 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.947808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.947820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.947880 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:27.948562 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.949058 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.950048 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.950253 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.950343 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.951095 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.951097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.951098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:27.951102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.951117 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:27.951121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.951169 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.951263 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.951296 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:26:27.951583 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.946957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38391 (local address 127.30.160.3:32943)
0808 02:26:27.947152 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.947185 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.947697 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.947977 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.947980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.947988 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:27.948467 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.948471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.949156 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.949157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.949941 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.949944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.950357 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.950358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.950505 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.950955 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:26:27.950955 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:27.950982 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:26:27.951004 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:27.951169 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:27.951175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:27.951178 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.951263 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.951360 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260808 02:26:27.952157 19995 heartbeater.cc:347] Connected to a master server at 127.30.160.62:38391
I20260808 02:26:27.952174 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 19860
I20260808 02:26:27.952239 19995 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:27.952328 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:27.952471 19995 heartbeater.cc:511] Master 127.30.160.62:38391 requested a full tablet report, sending...
I20260808 02:26:27.952543 31360 external_mini_cluster.cc:1428] Setting key 58711cade49f88cd430e525d3f955cb3
I20260808 02:26:27.952891 19519 ts_manager.cc:194] Registered new tserver with Master: 6de14058ef0c46fa8bf357acfcf8ad6f (127.30.160.3:43199)
I20260808 02:26:27.953374 19519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.160.3:32943
I20260808 02:26:27.953413 31360 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:27.957320 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.954710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.30.160.62:38391)
0808 02:26:27.954777 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.954780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.954874 (+ 94us) server_negotiation.cc:407] Connection header received
0808 02:26:27.954907 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.954907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.954915 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.954955 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:27.955088 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.955514 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.956269 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.956448 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.956500 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.957114 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:27.957114 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:27.957115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:27.957118 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:27.957127 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:27.957130 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:27.957167 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.957204 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.957218 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:26:27.958809 19519 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:27.960316 19519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:27.966368 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.962491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.30.160.3:43199)
0808 02:26:27.962655 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.962660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.962670 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:27.962744 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.962746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.962757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.962850 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.963010 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.964051 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.965439 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.965966 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.966048 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.966146 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.966258 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.966292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260808 02:26:27.966387 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.962340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43199 (local address 127.0.0.1:48242)
0808 02:26:27.962610 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.962623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.962676 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.962861 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.962865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.962867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.962937 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.962939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.964159 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.964160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.965348 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.965351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.966062 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.966063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.966154 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.966167 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.966247 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.966317 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":92,"threads_started":1}
I20260808 02:26:27.966547 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.962333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38119 (local address 127.0.0.1:55764)
0808 02:26:27.962484 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.962524 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.962587 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.963249 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.963250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.963252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.963287 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.963289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.964353 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.964354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.965405 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.965408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.966346 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.966346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.966401 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.966405 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.966435 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.966475 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":49,"threads_started":1}
I20260808 02:26:27.966924 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.962514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:38761 (local address 127.0.0.1:57132)
0808 02:26:27.962896 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:27.962906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:27.962962 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:27.963493 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:27.963494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:27.963496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:27.963534 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.963536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.964781 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.964782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.965793 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:27.965797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.966723 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.966724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:27.966780 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.966784 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:26:27.966828 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 02:26:27.966865 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":55,"threads_started":1}
I20260808 02:26:27.966969 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.962421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.30.160.2:38119)
0808 02:26:27.963111 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.963114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.963122 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:27.963152 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.963153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.963165 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.963252 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.963355 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.964266 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.965504 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.966257 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.966715 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.966824 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.966850 (+ 26us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.966884 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":91,"threads_started":1}
I20260808 02:26:27.967159 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:27.962709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.30.160.1:38761)
0808 02:26:27.962906 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:27.962910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:27.962953 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:26:27.963326 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:27.963328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:27.963348 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:27.963430 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:27.963608 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.964694 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.965881 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:27.966650 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:27.966721 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:27.967018 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:27.967045 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 02:26:27.967083 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260808 02:26:27.967357 19927 tablet_service.cc:1467] Processing CreateTablet for tablet 8222db55472b42eebb1d649afd763726 (DEFAULT_TABLE table=test-table [id=eae8686ff8494f05a2e4452bdafe0170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:27.967533 19788 tablet_service.cc:1467] Processing CreateTablet for tablet 8222db55472b42eebb1d649afd763726 (DEFAULT_TABLE table=test-table [id=eae8686ff8494f05a2e4452bdafe0170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:27.967739 19927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8222db55472b42eebb1d649afd763726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.967973 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8222db55472b42eebb1d649afd763726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.970290 20010 tablet_bootstrap.cc:492] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Bootstrap starting.
I20260808 02:26:27.970394 20011 tablet_bootstrap.cc:492] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Bootstrap starting.
I20260808 02:26:27.971088 20010 tablet_bootstrap.cc:654] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.971005 19650 tablet_service.cc:1467] Processing CreateTablet for tablet 8222db55472b42eebb1d649afd763726 (DEFAULT_TABLE table=test-table [id=eae8686ff8494f05a2e4452bdafe0170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:27.971333 20011 tablet_bootstrap.cc:654] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.971359 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8222db55472b42eebb1d649afd763726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:27.971417 20010 log.cc:824] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.971697 20011 log.cc:824] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.972225 20010 tablet_bootstrap.cc:492] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: No bootstrap required, opened a new log
I20260808 02:26:27.972301 20010 ts_tablet_manager.cc:1397] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:27.972427 20011 tablet_bootstrap.cc:492] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: No bootstrap required, opened a new log
I20260808 02:26:27.972512 20011 ts_tablet_manager.cc:1397] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260808 02:26:27.974547 20011 raft_consensus.cc:348] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:27.974745 20011 raft_consensus.cc:374] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.974792 20011 raft_consensus.cc:729] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f131811e9524d88bdfc011239f09b85, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.974922 20014 tablet_bootstrap.cc:492] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177: Bootstrap starting.
I20260808 02:26:27.974924 20011 consensus_queue.cc:260] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:27.975208 20011 ts_tablet_manager.cc:1428] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:26:27.975142 20010 raft_consensus.cc:348] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:27.975281 20010 raft_consensus.cc:374] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.975323 20010 raft_consensus.cc:729] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de14058ef0c46fa8bf357acfcf8ad6f, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.975354 19857 heartbeater.cc:503] Master 127.30.160.62:38391 was elected leader, sending a full tablet report...
I20260808 02:26:27.975458 20010 consensus_queue.cc:260] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:27.975628 20014 tablet_bootstrap.cc:654] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:27.975711 20010 ts_tablet_manager.cc:1428] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:26:27.975836 19995 heartbeater.cc:503] Master 127.30.160.62:38391 was elected leader, sending a full tablet report...
I20260808 02:26:27.975991 20014 log.cc:824] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:27.976763 20014 tablet_bootstrap.cc:492] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177: No bootstrap required, opened a new log
I20260808 02:26:27.976858 20014 ts_tablet_manager.cc:1397] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:26:27.979171 20014 raft_consensus.cc:348] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:27.979283 20014 raft_consensus.cc:374] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:27.979306 20014 raft_consensus.cc:729] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1d3d7980594f1b93af6c454704b177, State: Initialized, Role: FOLLOWER
I20260808 02:26:27.979414 20014 consensus_queue.cc:260] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:27.979717 20014 ts_tablet_manager.cc:1428] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:26:27.979795 19719 heartbeater.cc:503] Master 127.30.160.62:38391 was elected leader, sending a full tablet report...
I20260808 02:26:28.036815 20019 raft_consensus.cc:482] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:28.036990 20019 raft_consensus.cc:504] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:28.037431 20019 leader_election.cc:288] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f131811e9524d88bdfc011239f09b85 (127.30.160.2:38119), 6de14058ef0c46fa8bf357acfcf8ad6f (127.30.160.3:43199)
I20260808 02:26:28.040236 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:28.037665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:36649 (local address 127.30.160.2:38119)
0808 02:26:28.037712 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:28.037714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:28.037724 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:28.037824 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:28.037825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:28.037834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:28.037911 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:28.038071 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.038547 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.039475 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.039844 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.039963 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:28.039972 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:28.040127 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:26:28.040168 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:26:28.040236 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:28.037606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.2:38119 (local address 127.30.160.1:36649)
0808 02:26:28.037667 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:28.037680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:28.037725 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:28.037914 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:28.037915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:28.037917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:28.038000 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:28.038002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.038630 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.038631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:28.039371 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:28.039373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.039962 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.039963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:28.040039 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:28.040042 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:28.040118 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:26:28.040164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:26:28.040331 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:28.037666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.160.1:38065 (local address 127.30.160.3:43199)
0808 02:26:28.037712 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:28.037714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:28.037873 (+ 159us) server_negotiation.cc:407] Connection header received
0808 02:26:28.037949 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:28.037950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:28.037960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:28.038012 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:28.038161 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.038622 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.039541 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.039969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.040049 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:28.040057 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:28.040215 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:26:28.040254 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:26:28.040548 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:28.037614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.3:43199 (local address 127.30.160.1:38065)
0808 02:26:28.037803 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:28.037812 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:28.037888 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:28.038014 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:28.038014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:28.038016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:28.038091 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:28.038092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.038705 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.038706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:28.039442 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:28.039443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.040049 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.040050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:28.040125 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:28.040128 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:26:28.040206 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:26:28.040244 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":13,"thread_start_us":92,"threads_started":1}
I20260808 02:26:28.040575 19810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222db55472b42eebb1d649afd763726" candidate_uuid: "dc1d3d7980594f1b93af6c454704b177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f131811e9524d88bdfc011239f09b85" is_pre_election: true
I20260808 02:26:28.040771 19810 raft_consensus.cc:2454] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1d3d7980594f1b93af6c454704b177 in term 0.
I20260808 02:26:28.040925 19948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222db55472b42eebb1d649afd763726" candidate_uuid: "dc1d3d7980594f1b93af6c454704b177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" is_pre_election: true
I20260808 02:26:28.041069 19948 raft_consensus.cc:2454] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1d3d7980594f1b93af6c454704b177 in term 0.
I20260808 02:26:28.041045 19600 leader_election.cc:302] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f131811e9524d88bdfc011239f09b85, dc1d3d7980594f1b93af6c454704b177; no voters:
I20260808 02:26:28.041237 20019 raft_consensus.cc:2781] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:28.041289 20019 raft_consensus.cc:482] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:28.041306 20019 raft_consensus.cc:3037] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:28.041978 20019 raft_consensus.cc:504] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:28.042115 20019 leader_election.cc:288] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [CANDIDATE]: Term 1 election: Requested vote from peers 1f131811e9524d88bdfc011239f09b85 (127.30.160.2:38119), 6de14058ef0c46fa8bf357acfcf8ad6f (127.30.160.3:43199)
I20260808 02:26:28.042279 19810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222db55472b42eebb1d649afd763726" candidate_uuid: "dc1d3d7980594f1b93af6c454704b177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f131811e9524d88bdfc011239f09b85"
I20260808 02:26:28.042306 19948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222db55472b42eebb1d649afd763726" candidate_uuid: "dc1d3d7980594f1b93af6c454704b177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f"
I20260808 02:26:28.042363 19810 raft_consensus.cc:3037] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:28.042373 19948 raft_consensus.cc:3037] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:28.043308 19948 raft_consensus.cc:2454] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1d3d7980594f1b93af6c454704b177 in term 1.
I20260808 02:26:28.043308 19810 raft_consensus.cc:2454] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1d3d7980594f1b93af6c454704b177 in term 1.
I20260808 02:26:28.043488 19602 leader_election.cc:302] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de14058ef0c46fa8bf357acfcf8ad6f, dc1d3d7980594f1b93af6c454704b177; no voters:
I20260808 02:26:28.043599 20019 raft_consensus.cc:2781] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:28.043745 20019 raft_consensus.cc:686] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [term 1 LEADER]: Becoming Leader. State: Replica: dc1d3d7980594f1b93af6c454704b177, State: Running, Role: LEADER
I20260808 02:26:28.043846 20019 consensus_queue.cc:237] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } }
I20260808 02:26:28.044764 19519 catalog_manager.cc:5495] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc1d3d7980594f1b93af6c454704b177 (127.30.160.1). New cstate: current_term: 1 leader_uuid: "dc1d3d7980594f1b93af6c454704b177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc1d3d7980594f1b93af6c454704b177" member_type: VOTER last_known_addr { host: "127.30.160.1" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 } health_report { overall_health: UNKNOWN } } }
W20260808 02:26:28.053414 19858 tablet.cc:2369] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:28.058190 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:28.055632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.30.160.1:38761)
0808 02:26:28.055670 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:28.055672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:28.055701 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:26:28.055784 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:28.055784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:28.055812 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:28.055871 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:28.056027 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.056634 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.057584 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:28.057761 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:28.057829 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:28.057940 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:28.058027 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:28.058080 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:28.058111 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 02:26:28.058126 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:28.059912 19810 raft_consensus.cc:1261] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 1 FOLLOWER]: Refusing update from remote peer dc1d3d7980594f1b93af6c454704b177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:28.059870 19948 raft_consensus.cc:1261] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 1 FOLLOWER]: Refusing update from remote peer dc1d3d7980594f1b93af6c454704b177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:28.060274 20019 consensus_queue.cc:1035] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f131811e9524d88bdfc011239f09b85" member_type: VOTER last_known_addr { host: "127.30.160.2" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:28.060415 20021 consensus_queue.cc:1035] T 8222db55472b42eebb1d649afd763726 P dc1d3d7980594f1b93af6c454704b177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6de14058ef0c46fa8bf357acfcf8ad6f" member_type: VOTER last_known_addr { host: "127.30.160.3" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:28.062606 20027 mvcc.cc:205] Tried to move back new op lower bound from 7316094927090593792 to 7316094927028543488. Current Snapshot: MvccSnapshot[applied={T|T < 7316094927090593792}]
I20260808 02:26:28.063228 20026 mvcc.cc:205] Tried to move back new op lower bound from 7316094927090593792 to 7316094927028543488. Current Snapshot: MvccSnapshot[applied={T|T < 7316094927090593792}]
I20260808 02:26:28.064857 19519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:28.064946 19519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:28.065701 19519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4aa6d31b7af471cb17e5e3b95a47767: Sending DeleteTablet for 3 replicas of tablet 8222db55472b42eebb1d649afd763726
I20260808 02:26:28.066030 19927 tablet_service.cc:1514] Processing DeleteTablet for tablet 8222db55472b42eebb1d649afd763726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:28 UTC) from {username='slave'} at 127.0.0.1:48242
I20260808 02:26:28.066064 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet 8222db55472b42eebb1d649afd763726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:28 UTC) from {username='slave'} at 127.0.0.1:55764
I20260808 02:26:28.066103 19650 tablet_service.cc:1514] Processing DeleteTablet for tablet 8222db55472b42eebb1d649afd763726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:28 UTC) from {username='slave'} at 127.0.0.1:57132
I20260808 02:26:28.066298 20034 tablet_replica.cc:330] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: stopping tablet replica
I20260808 02:26:28.066327 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19583
I20260808 02:26:28.066391 20034 raft_consensus.cc:2229] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:28.066493 20034 raft_consensus.cc:2258] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:28.066903 20034 ts_tablet_manager.cc:1905] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:28.068671 20034 ts_tablet_manager.cc:1918] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:28.068742 20034 log.cc:1196] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-2/wal/wals/8222db55472b42eebb1d649afd763726
I20260808 02:26:28.068964 20034 ts_tablet_manager.cc:1939] T 8222db55472b42eebb1d649afd763726 P 6de14058ef0c46fa8bf357acfcf8ad6f: Deleting consensus metadata
I20260808 02:26:28.068984 20035 tablet_replica.cc:330] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: stopping tablet replica
I20260808 02:26:28.069108 20035 raft_consensus.cc:2229] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:28.069204 20035 raft_consensus.cc:2258] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:28.069428 19503 catalog_manager.cc:4841] TS 6de14058ef0c46fa8bf357acfcf8ad6f (127.30.160.3:43199): tablet 8222db55472b42eebb1d649afd763726 (table test-table [id=eae8686ff8494f05a2e4452bdafe0170]) successfully deleted
I20260808 02:26:28.069653 20035 ts_tablet_manager.cc:1905] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:28.071020 20035 ts_tablet_manager.cc:1918] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:28.071110 20035 log.cc:1196] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Deleting WAL directory at /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155746594248-31360-0/minicluster-data/ts-1/wal/wals/8222db55472b42eebb1d649afd763726
I20260808 02:26:28.071341 20035 ts_tablet_manager.cc:1939] T 8222db55472b42eebb1d649afd763726 P 1f131811e9524d88bdfc011239f09b85: Deleting consensus metadata
I20260808 02:26:28.071736 19501 catalog_manager.cc:4841] TS 1f131811e9524d88bdfc011239f09b85 (127.30.160.2:38119): tablet 8222db55472b42eebb1d649afd763726 (table test-table [id=eae8686ff8494f05a2e4452bdafe0170]) successfully deleted
W20260808 02:26:28.074138 19501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.160.1:38761 (error 108)
I20260808 02:26:28.074275 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19723
I20260808 02:26:28.075390 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:28.075221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.1:38761 (local address 127.0.0.1:57156)
0808 02:26:28.075290 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:28.075348 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.1:38761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 02:26:28.075534 19501 catalog_manager.cc:4584] TS dc1d3d7980594f1b93af6c454704b177 (127.30.160.1:38761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8222db55472b42eebb1d649afd763726: Network error: Client connection negotiation failed: client connection to 127.30.160.1:38761: connect: Connection refused (error 111)
I20260808 02:26:28.080715 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19860
I20260808 02:26:28.086079 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 19484
2026-08-08T02:26:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (739 ms)
[----------] 6 tests from EncryptionPolicyTest (8663 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:28 dist-test-slave-zkpd krb5kdc[20040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:28 dist-test-slave-zkpd krb5kdc[20040](info): set up 2 sockets
Aug 08 02:26:28 dist-test-slave-zkpd krb5kdc[20040](info): commencing operation
krb5kdc: starting...
W20260808 02:26:29.139498 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:29 dist-test-slave-zkpd krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:29.258272 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:29.340338 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:29.340543 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:29.340565 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:29.341720 20053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:29.341764 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:29.341785 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:29.341840 20053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:29.341857 20053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:29.343336 20053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:29.343581 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:29.343815 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:29.348819 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:29.348835 20053 server_base.cc:1034] running on GCE node
W20260808 02:26:29.348817 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:29.348838 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:29.349236 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:29.349536 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:29.350703 20053 hybrid_clock.cc:648] HybridClock initialized: now 1786155989350689 us; error 25 us; skew 500 ppm
Aug 08 02:26:29 dist-test-slave-zkpd krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:29.353183 20053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:29.353653 20053 webserver.cc:460] Webserver started at http://127.0.0.1:46441/ using document root <none> and password file <none>
I20260808 02:26:29.353899 20053 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:29.353967 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:29.354125 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:29.355187 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "b466c53a33dc44f88aaa2ca42085ee17"
format_stamp: "Formatted at 2026-08-08 02:26:29 on dist-test-slave-zkpd"
server_key: "8b8f5df323e4a5948662f994aa41152e"
server_key_iv: "cfd4fbdcac84bd06e698e1ffa4a68746"
server_key_version: "encryptionkey@0"
I20260808 02:26:29.355558 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "b466c53a33dc44f88aaa2ca42085ee17"
format_stamp: "Formatted at 2026-08-08 02:26:29 on dist-test-slave-zkpd"
server_key: "8b8f5df323e4a5948662f994aa41152e"
server_key_iv: "cfd4fbdcac84bd06e698e1ffa4a68746"
server_key_version: "encryptionkey@0"
I20260808 02:26:29.356951 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:29.357664 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:29.357838 20053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:29.357941 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "b466c53a33dc44f88aaa2ca42085ee17"
format_stamp: "Formatted at 2026-08-08 02:26:29 on dist-test-slave-zkpd"
server_key: "8b8f5df323e4a5948662f994aa41152e"
server_key_iv: "cfd4fbdcac84bd06e698e1ffa4a68746"
server_key_version: "encryptionkey@0"
I20260808 02:26:29.358023 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:29.377667 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:29.379503 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:29.379664 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:29.383862 20053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36181
I20260808 02:26:29.383988 20123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36181 every 8 connection(s)
I20260808 02:26:29.384339 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:29.385250 20124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:29.387566 20124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17: Bootstrap starting.
I20260808 02:26:29.388172 20124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:29.388486 20124 log.cc:824] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:29.389178 20124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17: No bootstrap required, opened a new log
I20260808 02:26:29.390781 20124 raft_consensus.cc:348] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466c53a33dc44f88aaa2ca42085ee17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260808 02:26:29.390915 20124 raft_consensus.cc:374] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:29.390937 20124 raft_consensus.cc:729] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b466c53a33dc44f88aaa2ca42085ee17, State: Initialized, Role: FOLLOWER
I20260808 02:26:29.391147 20124 consensus_queue.cc:260] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466c53a33dc44f88aaa2ca42085ee17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260808 02:26:29.391232 20124 raft_consensus.cc:388] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:29.391270 20124 raft_consensus.cc:482] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:29.391319 20124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:29.391810 20124 raft_consensus.cc:504] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466c53a33dc44f88aaa2ca42085ee17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260808 02:26:29.391939 20124 leader_election.cc:302] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b466c53a33dc44f88aaa2ca42085ee17; no voters:
I20260808 02:26:29.392124 20124 leader_election.cc:288] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:29.392184 20127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:29.392371 20124 sys_catalog.cc:564] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:29.392386 20127 raft_consensus.cc:686] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [term 1 LEADER]: Becoming Leader. State: Replica: b466c53a33dc44f88aaa2ca42085ee17, State: Running, Role: LEADER
I20260808 02:26:29.392491 20127 consensus_queue.cc:237] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466c53a33dc44f88aaa2ca42085ee17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260808 02:26:29.392911 20128 sys_catalog.cc:455] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b466c53a33dc44f88aaa2ca42085ee17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466c53a33dc44f88aaa2ca42085ee17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } } }
I20260808 02:26:29.392999 20128 sys_catalog.cc:458] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:29.393169 20129 sys_catalog.cc:455] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b466c53a33dc44f88aaa2ca42085ee17. Latest consensus state: current_term: 1 leader_uuid: "b466c53a33dc44f88aaa2ca42085ee17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466c53a33dc44f88aaa2ca42085ee17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } } }
I20260808 02:26:29.393265 20129 sys_catalog.cc:458] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:29.393265 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20053
I20260808 02:26:29.393409 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:29.393582 31360 external_mini_cluster.cc:1428] Setting key a1a577d909ce8fbeac48d3be806b3f04
W20260808 02:26:29.394973 20145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:29.395063 20145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:29.395144 20135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:29.395308 20135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:29.396803 20135 catalog_manager.cc:1347] Generated new cluster ID: 071e8b4166ce4909a84ed3ba674c7cab
I20260808 02:26:29.396860 20135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:29 dist-test-slave-zkpd krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:26:29.402285 20135 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:29.402953 20135 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:29.404147 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:29.394171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35467 (local address 127.0.0.1:36181)
0808 02:26:29.394397 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:29.394405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:29.394429 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:26:29.394554 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:29.394561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:29.394622 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:29.394797 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:29.395571 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:29.396844 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:29.397647 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:29.398065 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:29.398179 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:29.400640 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:29.400644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:29.400645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:29.400657 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:29.402506 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:29.402902 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:29.402905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:29.402905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:29.403016 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:29.403281 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:29.403283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:29.403283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:29.403522 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:29.403713 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:29.403810 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 02:26:29.403932 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260808 02:26:29.405068 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:29.409232 20135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b466c53a33dc44f88aaa2ca42085ee17: Generated new TSK 0
I20260808 02:26:29.409396 20135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:29.416061 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:29.409132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60743 (local address 127.0.0.1:36181)
0808 02:26:29.409537 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:29.409538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:29.409818 (+ 280us) server_negotiation.cc:407] Connection header received
0808 02:26:29.409845 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:29.409846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:29.409858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:29.409919 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:29.410412 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:29.411206 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:29.411915 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:29.412248 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:29.412321 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:29.413437 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:29.413438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:29.413439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:29.413445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:29.414915 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:29.415267 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:29.415268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:29.415268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:29.415329 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:29.415535 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:29.415536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:29.415536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:29.415659 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:29.415769 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:29.415868 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 02:26:29.415971 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:26:29.417886 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20053
W20260808 02:26:29.424970 20038 connection.cc:535] client connection to 127.0.0.1:36181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36181): Connection reset by peer (error 104)
2026-08-08T02:26:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:29 dist-test-slave-zkpd krb5kdc[20156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:29 dist-test-slave-zkpd krb5kdc[20156](info): set up 2 sockets
Aug 08 02:26:29 dist-test-slave-zkpd krb5kdc[20156](info): commencing operation
krb5kdc: starting...
W20260808 02:26:30.484149 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:30 dist-test-slave-zkpd krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:30.603246 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:30.684028 20171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:30.684257 20171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:30.684293 20171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:30.685429 20171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:30.685497 20171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:30.685524 20171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:30.685559 20171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:30.685583 20171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:30.687042 20171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:30.687309 20171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:30.687531 20171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:30.692289 20180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:30.692298 20177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:30.692370 20178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:30.692422 20171 server_base.cc:1034] running on GCE node
I20260808 02:26:30.692777 20171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:30.693065 20171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:30.694231 20171 hybrid_clock.cc:648] HybridClock initialized: now 1786155990694209 us; error 34 us; skew 500 ppm
Aug 08 02:26:30 dist-test-slave-zkpd krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:30.700176 20171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:30.700627 20171 webserver.cc:460] Webserver started at http://127.0.0.1:41033/ using document root <none> and password file <none>
I20260808 02:26:30.700856 20171 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:30.700924 20171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:30.701074 20171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:30.702137 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "736d6580d11446bc8e2a8de1dccc8e14"
format_stamp: "Formatted at 2026-08-08 02:26:30 on dist-test-slave-zkpd"
server_key: "38eebcaa0eaaa78cc246e7e85e402604"
server_key_iv: "9123eafcf164d9c8f03ca32f8ae29d2a"
server_key_version: "encryptionkey@0"
I20260808 02:26:30.702512 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "736d6580d11446bc8e2a8de1dccc8e14"
format_stamp: "Formatted at 2026-08-08 02:26:30 on dist-test-slave-zkpd"
server_key: "38eebcaa0eaaa78cc246e7e85e402604"
server_key_iv: "9123eafcf164d9c8f03ca32f8ae29d2a"
server_key_version: "encryptionkey@0"
I20260808 02:26:30.703837 20171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:30.704600 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:30.704771 20171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:30.704879 20171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "736d6580d11446bc8e2a8de1dccc8e14"
format_stamp: "Formatted at 2026-08-08 02:26:30 on dist-test-slave-zkpd"
server_key: "38eebcaa0eaaa78cc246e7e85e402604"
server_key_iv: "9123eafcf164d9c8f03ca32f8ae29d2a"
server_key_version: "encryptionkey@0"
I20260808 02:26:30.704953 20171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:30.738432 20171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:30.740844 20171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:30.741005 20171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:30.744923 20171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43809
I20260808 02:26:30.744962 20242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43809 every 8 connection(s)
I20260808 02:26:30.745334 20171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:30.746232 20243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:30.748301 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20171
I20260808 02:26:30.748441 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:30.748643 31360 external_mini_cluster.cc:1428] Setting key 12c4968024808da6e86ccdc2746a0c2e
I20260808 02:26:30.748816 20243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14: Bootstrap starting.
I20260808 02:26:30.749563 20243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:30.749876 20243 log.cc:824] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:30.750680 20243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14: No bootstrap required, opened a new log
I20260808 02:26:30.753006 20243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736d6580d11446bc8e2a8de1dccc8e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43809 } }
I20260808 02:26:30.753172 20243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:30.753206 20243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736d6580d11446bc8e2a8de1dccc8e14, State: Initialized, Role: FOLLOWER
I20260808 02:26:30.753350 20243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736d6580d11446bc8e2a8de1dccc8e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43809 } }
I20260808 02:26:30.753435 20243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:30.753468 20243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:30.753545 20243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:30.754159 20243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736d6580d11446bc8e2a8de1dccc8e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43809 } }
I20260808 02:26:30.754276 20243 leader_election.cc:302] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736d6580d11446bc8e2a8de1dccc8e14; no voters:
I20260808 02:26:30.754452 20243 leader_election.cc:288] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:30.754637 20248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:30.754709 20243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:30.754828 20248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [term 1 LEADER]: Becoming Leader. State: Replica: 736d6580d11446bc8e2a8de1dccc8e14, State: Running, Role: LEADER
I20260808 02:26:30.755007 20248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736d6580d11446bc8e2a8de1dccc8e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43809 } }
I20260808 02:26:30.755038 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:30.749213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42789 (local address 127.0.0.1:43809)
0808 02:26:30.749400 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:30.749408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:30.749500 (+ 92us) server_negotiation.cc:407] Connection header received
0808 02:26:30.750116 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:30.750123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:30.750171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:30.750283 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:30.750527 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:30.751926 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:30.752653 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:30.753278 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:30.753388 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:30.754108 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:30.754112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:30.754112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:30.754317 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:30.754356 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:30.754372 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:30.754462 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:30.754588 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 02:26:30.754619 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260808 02:26:30.755431 20249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736d6580d11446bc8e2a8de1dccc8e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736d6580d11446bc8e2a8de1dccc8e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43809 } } }
I20260808 02:26:30.755528 20249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:30.755679 20250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736d6580d11446bc8e2a8de1dccc8e14. Latest consensus state: current_term: 1 leader_uuid: "736d6580d11446bc8e2a8de1dccc8e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736d6580d11446bc8e2a8de1dccc8e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43809 } } }
I20260808 02:26:30.756067 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:30.755837 20257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:30.756307 20250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:30.756812 20257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:30.758600 20257 catalog_manager.cc:1347] Generated new cluster ID: fce3bbd121a5451191f3d2c6809c0b92
I20260808 02:26:30.758643 20257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:30.761396 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:30.758027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43541 (local address 127.0.0.1:43809)
0808 02:26:30.758092 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:30.758094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:30.758102 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:30.758122 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:30.758122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:30.758133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:30.758196 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:30.758370 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:30.759071 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:30.760121 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:30.760411 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:30.760475 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:30.761136 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:30.761137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:30.761137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:30.761141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:30.761155 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:30.761159 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:30.761216 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:30.761319 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:26:30.761328 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:26:30.765486 20257 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:30.765980 20257 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:30.769469 20257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736d6580d11446bc8e2a8de1dccc8e14: Generated new TSK 0
I20260808 02:26:30.769649 20257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:30.773995 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20171
W20260808 02:26:30.779799 20154 connection.cc:535] client connection to 127.0.0.1:43809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43809): Connection reset by peer (error 104)
2026-08-08T02:26:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:30 dist-test-slave-zkpd krb5kdc[20276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:30 dist-test-slave-zkpd krb5kdc[20276](info): set up 2 sockets
Aug 08 02:26:30 dist-test-slave-zkpd krb5kdc[20276](info): commencing operation
krb5kdc: starting...
W20260808 02:26:31.828878 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:31 dist-test-slave-zkpd krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:31.952771 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:32.035465 20290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:32.035666 20290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:32.035691 20290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:32.036788 20290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:32.036824 20290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:32.036839 20290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:32.036850 20290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:32.036862 20290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:32.038255 20290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:32.038436 20290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:32.038597 20290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:32.043442 20295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:32.043447 20298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:32.043480 20290 server_base.cc:1034] running on GCE node
W20260808 02:26:32.043462 20296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:32.044014 20290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:32.044317 20290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:32.045485 20290 hybrid_clock.cc:648] HybridClock initialized: now 1786155992045484 us; error 41 us; skew 500 ppm
Aug 08 02:26:32 dist-test-slave-zkpd krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:32.048055 20290 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:32.048525 20290 webserver.cc:460] Webserver started at http://127.0.0.1:45213/ using document root <none> and password file <none>
I20260808 02:26:32.048784 20290 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:32.048861 20290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:32.049021 20290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:32.050048 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "cc0dc2142db0401d8f66bae03c7001c0"
format_stamp: "Formatted at 2026-08-08 02:26:32 on dist-test-slave-zkpd"
server_key: "3019080bf7bafb6e845329612ae9a9f7"
server_key_iv: "536b7cdf4f0d7a1a7c90542e0cb399a9"
server_key_version: "encryptionkey@0"
I20260808 02:26:32.050421 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "cc0dc2142db0401d8f66bae03c7001c0"
format_stamp: "Formatted at 2026-08-08 02:26:32 on dist-test-slave-zkpd"
server_key: "3019080bf7bafb6e845329612ae9a9f7"
server_key_iv: "536b7cdf4f0d7a1a7c90542e0cb399a9"
server_key_version: "encryptionkey@0"
I20260808 02:26:32.051736 20290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:32.052631 20305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:32.052819 20290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:32.052927 20290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "cc0dc2142db0401d8f66bae03c7001c0"
format_stamp: "Formatted at 2026-08-08 02:26:32 on dist-test-slave-zkpd"
server_key: "3019080bf7bafb6e845329612ae9a9f7"
server_key_iv: "536b7cdf4f0d7a1a7c90542e0cb399a9"
server_key_version: "encryptionkey@0"
I20260808 02:26:32.053011 20290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:32.056357 20290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:32.058604 20290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:32.058763 20290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:32.062805 20290 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37237
I20260808 02:26:32.062830 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37237 every 8 connection(s)
I20260808 02:26:32.063153 20290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:32.064178 20362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:32.066361 20362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0: Bootstrap starting.
I20260808 02:26:32.066934 20362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:32.067216 20362 log.cc:824] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:32.067529 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20290
I20260808 02:26:32.067675 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:32.067881 31360 external_mini_cluster.cc:1428] Setting key 1a332221dd90d144ae79034b00c383dd
I20260808 02:26:32.068078 20362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0: No bootstrap required, opened a new log
I20260808 02:26:32.069864 20362 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0dc2142db0401d8f66bae03c7001c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260808 02:26:32.070043 20362 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:32.070128 20362 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0dc2142db0401d8f66bae03c7001c0, State: Initialized, Role: FOLLOWER
I20260808 02:26:32.070269 20362 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0dc2142db0401d8f66bae03c7001c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260808 02:26:32.070358 20362 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:32.070381 20362 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:32.070479 20362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:32.071218 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:32.068460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43675 (local address 127.0.0.1:37237)
0808 02:26:32.068694 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:32.068702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:32.068728 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:26:32.069847 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:32.069853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:32.069879 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:32.069992 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:32.070737 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:32.070740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:32.070740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:32.070744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:32.070780 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:32.070793 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:32.070848 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:32.070951 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:26:32.070983 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260808 02:26:32.071658 20362 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0dc2142db0401d8f66bae03c7001c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260808 02:26:32.071768 20362 leader_election.cc:302] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0dc2142db0401d8f66bae03c7001c0; no voters:
I20260808 02:26:32.071933 20362 leader_election.cc:288] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:32.071954 20367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:32.072119 20367 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [term 1 LEADER]: Becoming Leader. State: Replica: cc0dc2142db0401d8f66bae03c7001c0, State: Running, Role: LEADER
I20260808 02:26:32.072193 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:32.072223 20362 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:32.072268 20367 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0dc2142db0401d8f66bae03c7001c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260808 02:26:32.072592 20367 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0dc2142db0401d8f66bae03c7001c0. Latest consensus state: current_term: 1 leader_uuid: "cc0dc2142db0401d8f66bae03c7001c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0dc2142db0401d8f66bae03c7001c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } } }
I20260808 02:26:32.072659 20367 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:32.072808 20368 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc0dc2142db0401d8f66bae03c7001c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0dc2142db0401d8f66bae03c7001c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } } }
I20260808 02:26:32.072870 20368 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:32.072973 20373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:32.073710 20373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:32.075321 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:32.074225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37191 (local address 127.0.0.1:37237)
0808 02:26:32.074259 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:32.074260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:32.074394 (+ 134us) server_negotiation.cc:407] Connection header received
0808 02:26:32.074461 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:32.074462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:32.074467 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:32.074536 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:32.075111 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:32.075112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:32.075113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:32.075117 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:32.075130 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:32.075134 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:32.075199 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:32.075247 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 02:26:32.075261 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 02:26:32.075608 20373 catalog_manager.cc:1347] Generated new cluster ID: e918a8f6ac6c42a0966238d1e6f07d23
I20260808 02:26:32.075652 20373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:32.090272 20373 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:32.090467 20373 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:32.115712 20373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc0dc2142db0401d8f66bae03c7001c0: Generated new TSK 0
I20260808 02:26:32.115960 20373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:32.123812 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20290
2026-08-08T02:26:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:32 dist-test-slave-zkpd krb5kdc[20397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:32 dist-test-slave-zkpd krb5kdc[20397](info): set up 2 sockets
Aug 08 02:26:32 dist-test-slave-zkpd krb5kdc[20397](info): commencing operation
krb5kdc: starting...
W20260808 02:26:33.180203 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:33 dist-test-slave-zkpd krb5kdc[20397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:33.302106 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:33.384850 20410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:33.385042 20410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:33.385066 20410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:33.386204 20410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:33.386250 20410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:33.386265 20410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:33.386278 20410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:33.386317 20410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:33.387820 20410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:33.388121 20410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:33.388388 20410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:33.393079 20415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:33.393079 20418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:33.393055 20416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:33.393245 20410 server_base.cc:1034] running on GCE node
I20260808 02:26:33.393523 20410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:33.393849 20410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:33.395030 20410 hybrid_clock.cc:648] HybridClock initialized: now 1786155993395006 us; error 42 us; skew 500 ppm
Aug 08 02:26:33 dist-test-slave-zkpd krb5kdc[20397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:33.397589 20410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:33.398043 20410 webserver.cc:460] Webserver started at http://127.0.0.1:34369/ using document root <none> and password file <none>
I20260808 02:26:33.398262 20410 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:33.398303 20410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:33.398391 20410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:33.399355 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "08c55b9e0af340c3b4c3cbd8c925cef4"
format_stamp: "Formatted at 2026-08-08 02:26:33 on dist-test-slave-zkpd"
server_key: "8309b562d3093aa01a269a05f74d9c57"
server_key_iv: "23ac2b60272426eb4d7f665aa3348315"
server_key_version: "encryptionkey@0"
I20260808 02:26:33.399646 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "08c55b9e0af340c3b4c3cbd8c925cef4"
format_stamp: "Formatted at 2026-08-08 02:26:33 on dist-test-slave-zkpd"
server_key: "8309b562d3093aa01a269a05f74d9c57"
server_key_iv: "23ac2b60272426eb4d7f665aa3348315"
server_key_version: "encryptionkey@0"
I20260808 02:26:33.400986 20410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:33.401710 20425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:33.401897 20410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:33.401993 20410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "08c55b9e0af340c3b4c3cbd8c925cef4"
format_stamp: "Formatted at 2026-08-08 02:26:33 on dist-test-slave-zkpd"
server_key: "8309b562d3093aa01a269a05f74d9c57"
server_key_iv: "23ac2b60272426eb4d7f665aa3348315"
server_key_version: "encryptionkey@0"
I20260808 02:26:33.402047 20410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:33.415709 20410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:33.417516 20410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:33.417621 20410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:33.421319 20410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33545
I20260808 02:26:33.421418 20481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33545 every 8 connection(s)
I20260808 02:26:33.421715 20410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:33.422601 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:33.424878 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4: Bootstrap starting.
I20260808 02:26:33.425462 20482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:33.425699 20482 log.cc:824] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:33.426364 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4: No bootstrap required, opened a new log
I20260808 02:26:33.427353 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20410
I20260808 02:26:33.427459 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:33.427662 31360 external_mini_cluster.cc:1428] Setting key a9239f48f923108a300cb02fdd67b67d
I20260808 02:26:33.428323 20482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260808 02:26:33.428519 20482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:33.428572 20482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c55b9e0af340c3b4c3cbd8c925cef4, State: Initialized, Role: FOLLOWER
I20260808 02:26:33.428707 20482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260808 02:26:33.428815 20482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:33.428853 20482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:33.428910 20482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:33.429952 20482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260808 02:26:33.430042 20482 leader_election.cc:302] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08c55b9e0af340c3b4c3cbd8c925cef4; no voters:
I20260808 02:26:33.430234 20482 leader_election.cc:288] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:33.430276 20488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:33.430449 20488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [term 1 LEADER]: Becoming Leader. State: Replica: 08c55b9e0af340c3b4c3cbd8c925cef4, State: Running, Role: LEADER
I20260808 02:26:33.430505 20482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:33.430553 20488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260808 02:26:33.430912 20489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } } }
I20260808 02:26:33.431003 20490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08c55b9e0af340c3b4c3cbd8c925cef4. Latest consensus state: current_term: 1 leader_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c55b9e0af340c3b4c3cbd8c925cef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } } }
I20260808 02:26:33.431036 20489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:33.431073 20490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:33.431223 20492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:33.431742 20492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:33.433499 20492 catalog_manager.cc:1347] Generated new cluster ID: feadd8a2a8854d15bba66f1c7afacce7
I20260808 02:26:33.433560 20492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:33 dist-test-slave-zkpd krb5kdc[20397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:26:33.438575 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:33.428233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55483 (local address 127.0.0.1:33545)
0808 02:26:33.428423 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:33.428431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:33.428455 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:26:33.428564 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:33.428571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:33.428626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:33.428767 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:33.429590 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:33.430813 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:33.431935 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:33.432628 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:33.432713 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:33.435224 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:33.435228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:33.435228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:33.435240 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:33.436990 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:33.437378 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:33.437381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:33.437381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:33.437496 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:33.437721 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:33.437722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:33.437723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:33.437967 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:33.438177 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:33.438244 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 02:26:33.438386 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260808 02:26:33.439482 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:33.450170 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:33.440872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39547 (local address 127.0.0.1:33545)
0808 02:26:33.440912 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:33.440914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:33.444053 (+ 3139us) server_negotiation.cc:407] Connection header received
0808 02:26:33.444126 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:33.444127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:33.444138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:33.444182 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:33.444705 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:33.445186 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:33.445898 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:33.446107 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:33.446187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:33.447365 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:33.447367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:33.447367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:33.447373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:33.448836 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:33.449207 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:33.449209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:33.449209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:33.449263 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:33.449493 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:33.449494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:33.449495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:33.449607 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:33.449724 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:33.449997 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 02:26:33.450073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:33.450278 20492 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:33.450970 20492 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:33.463182 20492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08c55b9e0af340c3b4c3cbd8c925cef4: Generated new TSK 0
I20260808 02:26:33.463413 20492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:33.469318 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20410
2026-08-08T02:26:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:33 dist-test-slave-zkpd krb5kdc[20515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:33 dist-test-slave-zkpd krb5kdc[20515](info): set up 2 sockets
Aug 08 02:26:33 dist-test-slave-zkpd krb5kdc[20515](info): commencing operation
krb5kdc: starting...
W20260808 02:26:34.554433 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:34 dist-test-slave-zkpd krb5kdc[20515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:34.674474 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:34.756501 20528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:34.756699 20528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:34.756721 20528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:34.757870 20528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:34.757913 20528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:34.757927 20528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:34.757941 20528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:34.757978 20528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:34.759562 20528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:34.759814 20528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:34.760066 20528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:34.767513 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:34.767529 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:34.767634 20534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:34.767724 20528 server_base.cc:1034] running on GCE node
I20260808 02:26:34.768013 20528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:34.768455 20528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:34.769618 20528 hybrid_clock.cc:648] HybridClock initialized: now 1786155994769583 us; error 39 us; skew 500 ppm
Aug 08 02:26:34 dist-test-slave-zkpd krb5kdc[20515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:34.772377 20528 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:34.772861 20528 webserver.cc:460] Webserver started at http://127.0.0.1:39559/ using document root <none> and password file <none>
I20260808 02:26:34.773093 20528 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:34.773159 20528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:34.773293 20528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:34.774370 20528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8"
format_stamp: "Formatted at 2026-08-08 02:26:34 on dist-test-slave-zkpd"
server_key: "5d5d76a1737b2ffe0e4d4e2b5ea794a9"
server_key_iv: "cf3f08456bfeef1c271ddbf75513e7f5"
server_key_version: "encryptionkey@0"
I20260808 02:26:34.774742 20528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8"
format_stamp: "Formatted at 2026-08-08 02:26:34 on dist-test-slave-zkpd"
server_key: "5d5d76a1737b2ffe0e4d4e2b5ea794a9"
server_key_iv: "cf3f08456bfeef1c271ddbf75513e7f5"
server_key_version: "encryptionkey@0"
I20260808 02:26:34.776090 20528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:34.776809 20545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:34.776985 20528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:34.777084 20528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8"
format_stamp: "Formatted at 2026-08-08 02:26:34 on dist-test-slave-zkpd"
server_key: "5d5d76a1737b2ffe0e4d4e2b5ea794a9"
server_key_iv: "cf3f08456bfeef1c271ddbf75513e7f5"
server_key_version: "encryptionkey@0"
I20260808 02:26:34.777170 20528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:34.789390 20528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:34.791711 20528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:34.791873 20528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:34.795816 20528 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38429
I20260808 02:26:34.795848 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38429 every 8 connection(s)
I20260808 02:26:34.796212 20528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:34.797041 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:34.799043 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20528
I20260808 02:26:34.799144 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:34.799315 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8: Bootstrap starting.
I20260808 02:26:34.799331 31360 external_mini_cluster.cc:1428] Setting key 77775c8b595105d424676401748dbe83
I20260808 02:26:34.800108 20600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:34.800433 20600 log.cc:824] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:34.801184 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8: No bootstrap required, opened a new log
I20260808 02:26:34.803423 20600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 02:26:34.803586 20600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:34.803619 20600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c4a9a30e12400aa2cc0c8dc22ac9f8, State: Initialized, Role: FOLLOWER
I20260808 02:26:34.803743 20600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 02:26:34.803817 20600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:34.803845 20600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:34.803917 20600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:34.804522 20600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 02:26:34.804610 20600 leader_election.cc:302] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c4a9a30e12400aa2cc0c8dc22ac9f8; no voters:
I20260808 02:26:34.804792 20600 leader_election.cc:288] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:34.804821 20605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:34.805017 20605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 53c4a9a30e12400aa2cc0c8dc22ac9f8, State: Running, Role: LEADER
I20260808 02:26:34.805058 20600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:34.805153 20605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } }
I20260808 02:26:34.805573 20605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c4a9a30e12400aa2cc0c8dc22ac9f8. Latest consensus state: current_term: 1 leader_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } } }
I20260808 02:26:34.805583 20606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c4a9a30e12400aa2cc0c8dc22ac9f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38429 } } }
I20260808 02:26:34.805738 20606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:34.805714 20605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:34.806190 20614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:34.806344 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:34.799877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36891 (local address 127.0.0.1:38429)
0808 02:26:34.800110 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:34.800120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:34.800154 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:26:34.800768 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:34.800776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:34.800823 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:34.800993 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:34.801174 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:34.802550 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:34.803316 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:34.803951 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:34.804072 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:34.804706 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:34.804710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:34.804710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:34.804930 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:34.804974 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:34.804992 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:34.805095 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:34.805338 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 02:26:34.805444 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260808 02:26:34.806802 20614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:34.807477 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:34.808291 20614 catalog_manager.cc:1347] Generated new cluster ID: cb748ba431c24c41b28d34c8034ea014
I20260808 02:26:34.808350 20614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:34.812481 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:34.808978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43621 (local address 127.0.0.1:38429)
0808 02:26:34.809039 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:34.809040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:34.809232 (+ 192us) server_negotiation.cc:407] Connection header received
0808 02:26:34.809287 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:34.809288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:34.809298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:34.809351 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:34.809480 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:34.810168 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:34.811183 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:34.811478 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:34.811537 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:34.812243 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:34.812244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:34.812244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:34.812249 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:34.812264 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:34.812268 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:34.812319 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:34.812388 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:26:34.812408 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:26:34.819260 20614 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:34.819756 20614 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:34.827992 20614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c4a9a30e12400aa2cc0c8dc22ac9f8: Generated new TSK 0
I20260808 02:26:34.828174 20614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:34.830611 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20528
2026-08-08T02:26:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:34 dist-test-slave-zkpd krb5kdc[20632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:34 dist-test-slave-zkpd krb5kdc[20632](info): set up 2 sockets
Aug 08 02:26:34 dist-test-slave-zkpd krb5kdc[20632](info): commencing operation
krb5kdc: starting...
W20260808 02:26:35.904089 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:35 dist-test-slave-zkpd krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:36.024861 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:36.106431 20647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:36.106647 20647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:36.106683 20647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:36.107779 20647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:36.107846 20647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:36.107868 20647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:36.107888 20647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:36.107905 20647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:36.109414 20647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:36.109690 20647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:36.109921 20647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:36.114729 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:36.114749 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:36.114809 20647 server_base.cc:1034] running on GCE node
W20260808 02:26:36.114805 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:36.115149 20647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:36.115481 20647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:36.116640 20647 hybrid_clock.cc:648] HybridClock initialized: now 1786155996116580 us; error 93 us; skew 500 ppm
Aug 08 02:26:36 dist-test-slave-zkpd krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:36.123411 20647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:36.123880 20647 webserver.cc:460] Webserver started at http://127.0.0.1:45699/ using document root <none> and password file <none>
I20260808 02:26:36.124171 20647 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:36.124243 20647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:36.124402 20647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:36.125469 20647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "fd2908becc1b48008195fd41e3d2ba8b"
format_stamp: "Formatted at 2026-08-08 02:26:36 on dist-test-slave-zkpd"
server_key: "3269a03d6a1450a46c9e19fe054bf247"
server_key_iv: "10455828302c7e1b792aad9e42f5096f"
server_key_version: "encryptionkey@0"
I20260808 02:26:36.125844 20647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "fd2908becc1b48008195fd41e3d2ba8b"
format_stamp: "Formatted at 2026-08-08 02:26:36 on dist-test-slave-zkpd"
server_key: "3269a03d6a1450a46c9e19fe054bf247"
server_key_iv: "10455828302c7e1b792aad9e42f5096f"
server_key_version: "encryptionkey@0"
I20260808 02:26:36.127197 20647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:36.127957 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:36.128140 20647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:26:36.128247 20647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "fd2908becc1b48008195fd41e3d2ba8b"
format_stamp: "Formatted at 2026-08-08 02:26:36 on dist-test-slave-zkpd"
server_key: "3269a03d6a1450a46c9e19fe054bf247"
server_key_iv: "10455828302c7e1b792aad9e42f5096f"
server_key_version: "encryptionkey@0"
I20260808 02:26:36.128336 20647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:36.131609 20647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:36.133466 20647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:36.133628 20647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:36.137713 20647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41639
I20260808 02:26:36.137737 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41639 every 8 connection(s)
I20260808 02:26:36.138078 20647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:36.139081 20720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:36.139513 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20647
I20260808 02:26:36.139638 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:36.139823 31360 external_mini_cluster.cc:1428] Setting key 18438a17403e7a8e46b433d42f61d86d
I20260808 02:26:36.142175 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b: Bootstrap starting.
I20260808 02:26:36.142323 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:36.140404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39401 (local address 127.0.0.1:41639)
0808 02:26:36.140591 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:36.140599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:36.140631 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:26:36.141034 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:36.141040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:36.141064 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:36.141177 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:36.141801 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:36.141804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:36.141805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:36.141808 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:36.141844 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:36.141853 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:36.141908 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:36.142022 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 02:26:36.142079 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260808 02:26:36.142940 20720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:36.143271 20720 log.cc:824] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:36.143316 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:36.144085 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b: No bootstrap required, opened a new log
I20260808 02:26:36.145929 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:36.144850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59557 (local address 127.0.0.1:41639)
0808 02:26:36.144888 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:36.144890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:36.144965 (+ 75us) server_negotiation.cc:407] Connection header received
0808 02:26:36.145038 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:36.145039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:36.145042 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:36.145081 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:36.145581 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:36.145582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:36.145582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:36.145586 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:36.145597 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:36.145600 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:36.145645 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:36.145725 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 02:26:36.145734 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:26:36.146693 20720 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2908becc1b48008195fd41e3d2ba8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41639 } }
I20260808 02:26:36.146839 20720 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:36.146885 20720 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2908becc1b48008195fd41e3d2ba8b, State: Initialized, Role: FOLLOWER
I20260808 02:26:36.146997 20720 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2908becc1b48008195fd41e3d2ba8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41639 } }
I20260808 02:26:36.147096 20720 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:36.147140 20720 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:36.147189 20720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:36.147703 20720 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2908becc1b48008195fd41e3d2ba8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41639 } }
I20260808 02:26:36.147821 20720 leader_election.cc:302] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd2908becc1b48008195fd41e3d2ba8b; no voters:
I20260808 02:26:36.147992 20720 leader_election.cc:288] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:36.148049 20730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:36.148205 20730 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [term 1 LEADER]: Becoming Leader. State: Replica: fd2908becc1b48008195fd41e3d2ba8b, State: Running, Role: LEADER
I20260808 02:26:36.148240 20720 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:36.148362 20730 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2908becc1b48008195fd41e3d2ba8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41639 } }
I20260808 02:26:36.148767 20731 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd2908becc1b48008195fd41e3d2ba8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2908becc1b48008195fd41e3d2ba8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41639 } } }
I20260808 02:26:36.148797 20732 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd2908becc1b48008195fd41e3d2ba8b. Latest consensus state: current_term: 1 leader_uuid: "fd2908becc1b48008195fd41e3d2ba8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2908becc1b48008195fd41e3d2ba8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41639 } } }
I20260808 02:26:36.148967 20732 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:36.148954 20731 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:36.149312 20739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:36.149809 20739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:36.151372 20739 catalog_manager.cc:1347] Generated new cluster ID: acc3461d72c749569d0a046acf0b4f3d
I20260808 02:26:36.151432 20739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:36.164193 20739 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:36.164350 20739 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:36.170477 20739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd2908becc1b48008195fd41e3d2ba8b: Generated new TSK 0
I20260808 02:26:36.170639 20739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:36.178656 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20647
2026-08-08T02:26:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:36 dist-test-slave-zkpd krb5kdc[20751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:36 dist-test-slave-zkpd krb5kdc[20751](info): set up 2 sockets
Aug 08 02:26:36 dist-test-slave-zkpd krb5kdc[20751](info): commencing operation
krb5kdc: starting...
W20260808 02:26:37.227538 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:37 dist-test-slave-zkpd krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:37.346252 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:37.426730 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:37.426959 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:37.427000 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:37.428138 20764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:37.428208 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:37.428233 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:37.428262 20764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:37.428294 20764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:37.429737 20764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:37.430001 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:37.430222 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:37.434983 20771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:37.435045 20772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:37.435230 20764 server_base.cc:1034] running on GCE node
W20260808 02:26:37.434983 20774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:37.435534 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:37.435850 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:37.437058 20764 hybrid_clock.cc:648] HybridClock initialized: now 1786155997437045 us; error 28 us; skew 500 ppm
Aug 08 02:26:37 dist-test-slave-zkpd krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:37.439379 20764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:37.439800 20764 webserver.cc:460] Webserver started at http://127.0.0.1:34905/ using document root <none> and password file <none>
I20260808 02:26:37.440052 20764 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:37.440099 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:37.440184 20764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:37.441123 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "4009e1799f19432d8f56c0acce21409d"
format_stamp: "Formatted at 2026-08-08 02:26:37 on dist-test-slave-zkpd"
server_key: "e59019de32938a7dbd5fd35eb3497c4e"
server_key_iv: "437334fe52a8a95378d7af0eac03aa11"
server_key_version: "encryptionkey@0"
I20260808 02:26:37.441412 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "4009e1799f19432d8f56c0acce21409d"
format_stamp: "Formatted at 2026-08-08 02:26:37 on dist-test-slave-zkpd"
server_key: "e59019de32938a7dbd5fd35eb3497c4e"
server_key_iv: "437334fe52a8a95378d7af0eac03aa11"
server_key_version: "encryptionkey@0"
I20260808 02:26:37.442608 20764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:37.443246 20781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:37.443416 20764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:37.443496 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "4009e1799f19432d8f56c0acce21409d"
format_stamp: "Formatted at 2026-08-08 02:26:37 on dist-test-slave-zkpd"
server_key: "e59019de32938a7dbd5fd35eb3497c4e"
server_key_iv: "437334fe52a8a95378d7af0eac03aa11"
server_key_version: "encryptionkey@0"
I20260808 02:26:37.443549 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:37.469170 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:37.471119 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:37.471289 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:37.475215 20764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40931
I20260808 02:26:37.475291 20835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40931 every 8 connection(s)
I20260808 02:26:37.475628 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:37.476519 20836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:37.478773 20836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d: Bootstrap starting.
I20260808 02:26:37.479359 20836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:37.479637 20836 log.cc:824] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:37.480399 20836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d: No bootstrap required, opened a new log
I20260808 02:26:37.480897 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20764
I20260808 02:26:37.481020 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:37.481216 31360 external_mini_cluster.cc:1428] Setting key cfba33f418b9a0579775f97499635664
I20260808 02:26:37.482458 20836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4009e1799f19432d8f56c0acce21409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260808 02:26:37.482613 20836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:37.482678 20836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4009e1799f19432d8f56c0acce21409d, State: Initialized, Role: FOLLOWER
I20260808 02:26:37.482853 20836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4009e1799f19432d8f56c0acce21409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260808 02:26:37.482949 20836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:37.482995 20836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:37.483043 20836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:37.483879 20836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4009e1799f19432d8f56c0acce21409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260808 02:26:37.484053 20836 leader_election.cc:302] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4009e1799f19432d8f56c0acce21409d; no voters:
I20260808 02:26:37.484257 20836 leader_election.cc:288] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:37.484278 20841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:37.484397 20841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [term 1 LEADER]: Becoming Leader. State: Replica: 4009e1799f19432d8f56c0acce21409d, State: Running, Role: LEADER
I20260808 02:26:37.484534 20836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:37.484539 20841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4009e1799f19432d8f56c0acce21409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260808 02:26:37.484982 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4009e1799f19432d8f56c0acce21409d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4009e1799f19432d8f56c0acce21409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } } }
I20260808 02:26:37.485074 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:37.484992 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4009e1799f19432d8f56c0acce21409d. Latest consensus state: current_term: 1 leader_uuid: "4009e1799f19432d8f56c0acce21409d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4009e1799f19432d8f56c0acce21409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } } }
I20260808 02:26:37.485118 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d [sys.catalog]: This master's current role is: LEADER
W20260808 02:26:37.486521 20858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:37.486577 20858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:37.486626 20848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:37.486810 20848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:37.488291 20848 catalog_manager.cc:1347] Generated new cluster ID: b0e7513b5da44ea98533698a2725a763
I20260808 02:26:37.488343 20848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:37 dist-test-slave-zkpd krb5kdc[20751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:26:37.492775 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:37.481724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:34721 (local address 127.0.0.1:40931)
0808 02:26:37.481926 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:37.481936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:37.481968 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:26:37.482440 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:37.482445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:37.482493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:37.482630 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:37.483375 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.484483 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.485561 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.486187 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.486288 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:37.488960 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:37.488964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:37.488965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:37.488976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:37.490990 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:37.491428 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:37.491431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:37.491432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:37.491575 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:37.491826 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:37.491827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:37.491827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:37.492166 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:37.492342 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:37.492445 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:26:37.492554 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260808 02:26:37.493817 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:37.511375 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:37.496228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:46391 (local address 127.0.0.1:40931)
0808 02:26:37.496264 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:37.496266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:37.504074 (+ 7808us) server_negotiation.cc:407] Connection header received
0808 02:26:37.504161 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:37.504162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:37.504175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:37.504233 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:37.504872 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.505602 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.506644 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.506968 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.507026 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:37.508540 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:37.508542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:37.508542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:37.508549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:37.510138 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:37.510564 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:37.510566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:37.510566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:37.510622 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:37.510857 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:37.510858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:37.510858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:37.510975 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:37.511081 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:37.511221 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 02:26:37.511285 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:37.520665 20848 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:37.521198 20848 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:37.526111 20848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4009e1799f19432d8f56c0acce21409d: Generated new TSK 0
I20260808 02:26:37.526305 20848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:37.534155 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20764
W20260808 02:26:37.540467 20749 connection.cc:535] client connection to 127.0.0.1:40931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40931): Connection reset by peer (error 104)
2026-08-08T02:26:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:37 dist-test-slave-zkpd krb5kdc[20868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:37 dist-test-slave-zkpd krb5kdc[20868](info): set up 2 sockets
Aug 08 02:26:37 dist-test-slave-zkpd krb5kdc[20868](info): commencing operation
krb5kdc: starting...
W20260808 02:26:38.583822 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:38 dist-test-slave-zkpd krb5kdc[20868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:38.701985 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:38.783528 20881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:38.783751 20881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:38.783774 20881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:38.784937 20881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:38.784979 20881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:38.784994 20881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:38.785007 20881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:38.785019 20881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:38.786478 20881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:38.786665 20881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:38.786835 20881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:38.791394 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:38.791385 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:38.791386 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:38.791503 20881 server_base.cc:1034] running on GCE node
I20260808 02:26:38.791869 20881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:38.792213 20881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:38.793363 20881 hybrid_clock.cc:648] HybridClock initialized: now 1786155998793351 us; error 28 us; skew 500 ppm
Aug 08 02:26:38 dist-test-slave-zkpd krb5kdc[20868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:38.796015 20881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:38.796463 20881 webserver.cc:460] Webserver started at http://127.0.0.1:37201/ using document root <none> and password file <none>
I20260808 02:26:38.796715 20881 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:38.796758 20881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:38.796932 20881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:38.797920 20881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "d025c0f6e4e349cf95f4847cd0efac06"
format_stamp: "Formatted at 2026-08-08 02:26:38 on dist-test-slave-zkpd"
server_key: "e92f2fd0466f1a2e7fde68d5bcff1238"
server_key_iv: "af1dbbab1f3665b3dd06eb1fdb918758"
server_key_version: "encryptionkey@0"
I20260808 02:26:38.798271 20881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "d025c0f6e4e349cf95f4847cd0efac06"
format_stamp: "Formatted at 2026-08-08 02:26:38 on dist-test-slave-zkpd"
server_key: "e92f2fd0466f1a2e7fde68d5bcff1238"
server_key_iv: "af1dbbab1f3665b3dd06eb1fdb918758"
server_key_version: "encryptionkey@0"
I20260808 02:26:38.799573 20881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:38.800407 20896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:38.800590 20881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:38.800690 20881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "d025c0f6e4e349cf95f4847cd0efac06"
format_stamp: "Formatted at 2026-08-08 02:26:38 on dist-test-slave-zkpd"
server_key: "e92f2fd0466f1a2e7fde68d5bcff1238"
server_key_iv: "af1dbbab1f3665b3dd06eb1fdb918758"
server_key_version: "encryptionkey@0"
I20260808 02:26:38.800774 20881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:38.838526 20881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:38.841002 20881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:38.841172 20881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:38.845141 20881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37829
I20260808 02:26:38.845181 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37829 every 8 connection(s)
I20260808 02:26:38.845561 20881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:38.846405 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:38.846920 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20881
I20260808 02:26:38.847011 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:38.847223 31360 external_mini_cluster.cc:1428] Setting key c30505fa6c45300455f442ff96d53812
I20260808 02:26:38.849299 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06: Bootstrap starting.
I20260808 02:26:38.849970 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:38.850277 20951 log.cc:824] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:38.850975 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06: No bootstrap required, opened a new log
I20260808 02:26:38.852650 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:38.847734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:40965 (local address 127.0.0.1:37829)
0808 02:26:38.847947 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:38.847962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:38.847987 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:26:38.848374 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:38.848380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:38.848426 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:38.848539 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:38.848794 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.849710 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.850756 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.851209 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.851305 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:38.852010 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:38.852014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:38.852014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:38.852167 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:38.852195 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:38.852205 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:38.852270 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:38.852426 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:26:38.852459 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260808 02:26:38.853302 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d025c0f6e4e349cf95f4847cd0efac06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37829 } }
I20260808 02:26:38.853457 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:38.853538 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d025c0f6e4e349cf95f4847cd0efac06, State: Initialized, Role: FOLLOWER
I20260808 02:26:38.853616 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:38.853704 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d025c0f6e4e349cf95f4847cd0efac06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37829 } }
I20260808 02:26:38.853785 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:38.853812 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:38.853848 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:38.854545 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d025c0f6e4e349cf95f4847cd0efac06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37829 } }
I20260808 02:26:38.854664 20951 leader_election.cc:302] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d025c0f6e4e349cf95f4847cd0efac06; no voters:
I20260808 02:26:38.854831 20951 leader_election.cc:288] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:38.854924 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:38.855077 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [term 1 LEADER]: Becoming Leader. State: Replica: d025c0f6e4e349cf95f4847cd0efac06, State: Running, Role: LEADER
I20260808 02:26:38.855080 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:38.855216 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d025c0f6e4e349cf95f4847cd0efac06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37829 } }
I20260808 02:26:38.857067 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d025c0f6e4e349cf95f4847cd0efac06. Latest consensus state: current_term: 1 leader_uuid: "d025c0f6e4e349cf95f4847cd0efac06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d025c0f6e4e349cf95f4847cd0efac06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37829 } } }
I20260808 02:26:38.857167 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [sys.catalog]: This master's current role is: LEADER
W20260808 02:26:38.857347 20979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:38.857435 20979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:38.857460 20964 sys_catalog.cc:455] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d025c0f6e4e349cf95f4847cd0efac06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d025c0f6e4e349cf95f4847cd0efac06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37829 } } }
I20260808 02:26:38.857559 20981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:38.857607 20964 sys_catalog.cc:458] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:38.857780 20981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:38.859438 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:38.855176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:49953 (local address 127.0.0.1:37829)
0808 02:26:38.855206 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:38.855208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:38.855472 (+ 264us) server_negotiation.cc:407] Connection header received
0808 02:26:38.855561 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:38.855561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:38.855571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:38.855616 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:38.855780 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.856551 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.857594 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.857889 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.857948 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:38.859136 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:38.859137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:38.859138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:38.859142 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:38.859156 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:38.859160 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:38.859208 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:38.859361 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:26:38.859371 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 02:26:38.859730 20981 catalog_manager.cc:1347] Generated new cluster ID: 565aa8dc55614185abc4b6bb4e638bc4
I20260808 02:26:38.859778 20981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:38.879076 20981 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:38.879783 20981 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:38.882714 20981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d025c0f6e4e349cf95f4847cd0efac06: Generated new TSK 0
I20260808 02:26:38.882920 20981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:38.888060 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20881
W20260808 02:26:38.894173 20866 connection.cc:535] client connection to 127.0.0.1:37829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37829): Connection reset by peer (error 104)
2026-08-08T02:26:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:38 dist-test-slave-zkpd krb5kdc[20985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:38 dist-test-slave-zkpd krb5kdc[20985](info): set up 2 sockets
Aug 08 02:26:38 dist-test-slave-zkpd krb5kdc[20985](info): commencing operation
krb5kdc: starting...
W20260808 02:26:39.939743 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:40 dist-test-slave-zkpd krb5kdc[20985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:40.059013 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:40.141211 20998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:40.141413 20998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:40.141435 20998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:40.142514 20998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:40.142550 20998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:40.142565 20998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:40.142577 20998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:40.142589 20998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:40.144016 20998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:40.144215 20998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:40.144382 20998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:40.149154 21007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:40.149159 21004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:40.149298 20998 server_base.cc:1034] running on GCE node
W20260808 02:26:40.149159 21005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:40.149652 20998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:40.149967 20998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:40.151131 20998 hybrid_clock.cc:648] HybridClock initialized: now 1786156000151116 us; error 41 us; skew 500 ppm
Aug 08 02:26:40 dist-test-slave-zkpd krb5kdc[20985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:40.153597 20998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:40.154054 20998 webserver.cc:460] Webserver started at http://127.0.0.1:46789/ using document root <none> and password file <none>
I20260808 02:26:40.154239 20998 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:40.154280 20998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:40.154372 20998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:40.155334 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "b684e24eed554b40b9f2a08f028c1bae"
format_stamp: "Formatted at 2026-08-08 02:26:40 on dist-test-slave-zkpd"
server_key: "4ef20748252feb690a4810392909e0eb"
server_key_iv: "8fb6c98b545796fac76b123c2455e1c7"
server_key_version: "encryptionkey@0"
I20260808 02:26:40.155618 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "b684e24eed554b40b9f2a08f028c1bae"
format_stamp: "Formatted at 2026-08-08 02:26:40 on dist-test-slave-zkpd"
server_key: "4ef20748252feb690a4810392909e0eb"
server_key_iv: "8fb6c98b545796fac76b123c2455e1c7"
server_key_version: "encryptionkey@0"
I20260808 02:26:40.156911 20998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:40.157596 21014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:40.157789 20998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:40.157879 20998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "b684e24eed554b40b9f2a08f028c1bae"
format_stamp: "Formatted at 2026-08-08 02:26:40 on dist-test-slave-zkpd"
server_key: "4ef20748252feb690a4810392909e0eb"
server_key_iv: "8fb6c98b545796fac76b123c2455e1c7"
server_key_version: "encryptionkey@0"
I20260808 02:26:40.157933 20998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:40.161190 20998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:40.162922 20998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:40.163030 20998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:40.167052 20998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42301
I20260808 02:26:40.167083 21070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42301 every 8 connection(s)
I20260808 02:26:40.167428 20998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:40.168459 21071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:40.170634 21071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae: Bootstrap starting.
I20260808 02:26:40.171289 21071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:40.171579 21071 log.cc:824] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:40.172408 21071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae: No bootstrap required, opened a new log
I20260808 02:26:40.173420 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 20998
I20260808 02:26:40.173519 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:40.173709 31360 external_mini_cluster.cc:1428] Setting key 64d82d620f05c14320623a130323cac1
I20260808 02:26:40.174453 21071 raft_consensus.cc:348] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } }
I20260808 02:26:40.174633 21071 raft_consensus.cc:374] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:40.174682 21071 raft_consensus.cc:729] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b684e24eed554b40b9f2a08f028c1bae, State: Initialized, Role: FOLLOWER
I20260808 02:26:40.174902 21071 consensus_queue.cc:260] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } }
I20260808 02:26:40.175267 21076 raft_consensus.cc:482] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:40.175359 21076 raft_consensus.cc:504] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } }
I20260808 02:26:40.175487 21076 leader_election.cc:302] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b684e24eed554b40b9f2a08f028c1bae; no voters:
I20260808 02:26:40.175050 21071 raft_consensus.cc:388] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:40.175678 21077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:40.175805 21077 raft_consensus.cc:482] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:40.175637 21076 leader_election.cc:288] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260808 02:26:40.175967 21077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:40.176115 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:40.174237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:46263 (local address 127.0.0.1:42301)
0808 02:26:40.174444 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:40.174451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:40.174475 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:26:40.174565 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:40.174570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:40.174592 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:40.174732 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:40.175469 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:40.175474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:40.175474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:40.175479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:40.175529 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:40.175544 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:40.175618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:40.175837 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 02:26:40.175885 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260808 02:26:40.176661 21077 raft_consensus.cc:504] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } }
I20260808 02:26:40.176764 21077 leader_election.cc:302] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b684e24eed554b40b9f2a08f028c1bae; no voters:
I20260808 02:26:40.176815 21077 leader_election.cc:288] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:40.176859 21071 raft_consensus.cc:482] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:40.176926 21071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:26:40.177033 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:40.177768 21071 raft_consensus.cc:504] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } }
I20260808 02:26:40.177891 21076 raft_consensus.cc:2741] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20260808 02:26:40.178217 21071 leader_election.cc:302] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b684e24eed554b40b9f2a08f028c1bae; no voters:
I20260808 02:26:40.178293 21071 leader_election.cc:288] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:26:40.178313 21076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:26:40.178480 21076 raft_consensus.cc:686] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [term 2 LEADER]: Becoming Leader. State: Replica: b684e24eed554b40b9f2a08f028c1bae, State: Running, Role: LEADER
I20260808 02:26:40.178489 21071 sys_catalog.cc:564] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:40.178586 21076 consensus_queue.cc:237] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } }
I20260808 02:26:40.178990 21077 sys_catalog.cc:455] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b684e24eed554b40b9f2a08f028c1bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } } }
I20260808 02:26:40.179119 21077 sys_catalog.cc:458] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:40.179209 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b684e24eed554b40b9f2a08f028c1bae. Latest consensus state: current_term: 2 leader_uuid: "b684e24eed554b40b9f2a08f028c1bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b684e24eed554b40b9f2a08f028c1bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42301 } } }
I20260808 02:26:40.179338 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:40.179850 21089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:40.180050 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:40.178385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58077 (local address 127.0.0.1:42301)
0808 02:26:40.178443 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:40.178446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:40.178584 (+ 138us) server_negotiation.cc:407] Connection header received
0808 02:26:40.178782 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:40.178783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:40.178788 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:40.178847 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:40.179760 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:40.179761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:40.179761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:40.179765 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:40.179782 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:40.179787 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:40.179841 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:40.179953 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:26:40.179968 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:26:40.180576 21089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:40.182047 21089 catalog_manager.cc:1347] Generated new cluster ID: d8e7e5251a934bc1b05d07f0eac9af28
I20260808 02:26:40.182104 21089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:40.191627 21089 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:40.191776 21089 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:40.197435 21089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b684e24eed554b40b9f2a08f028c1bae: Generated new TSK 0
I20260808 02:26:40.197598 21089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:40.202791 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 20998
2026-08-08T02:26:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:40 dist-test-slave-zkpd krb5kdc[21104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:40 dist-test-slave-zkpd krb5kdc[21104](info): set up 2 sockets
Aug 08 02:26:40 dist-test-slave-zkpd krb5kdc[21104](info): commencing operation
krb5kdc: starting...
W20260808 02:26:41.260843 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:41 dist-test-slave-zkpd krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:41.380858 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:41.465036 21117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:41.465274 21117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:41.465313 21117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:41.466938 21117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:41.467013 21117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:41.467041 21117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:41.467090 21117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:41.467121 21117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:41.469135 21117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:41.469376 21117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:41.469620 21117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:41.474465 21122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:41.474514 21123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:41.474772 21117 server_base.cc:1034] running on GCE node
W20260808 02:26:41.474484 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:41.475075 21117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:41.475446 21117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:41.476622 21117 hybrid_clock.cc:648] HybridClock initialized: now 1786156001476617 us; error 36 us; skew 500 ppm
Aug 08 02:26:41 dist-test-slave-zkpd krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:41.483825 21117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:41.484311 21117 webserver.cc:460] Webserver started at http://127.0.0.1:42315/ using document root <none> and password file <none>
I20260808 02:26:41.484535 21117 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:41.484575 21117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:41.484663 21117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:41.485713 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "8f38c63c8c924bd594592f73a7459c1c"
format_stamp: "Formatted at 2026-08-08 02:26:41 on dist-test-slave-zkpd"
server_key: "8e611429b17204d63aa4c66ef27c3d2d"
server_key_iv: "cebb5e8b901e1b87cfbaeba969d8d064"
server_key_version: "encryptionkey@0"
I20260808 02:26:41.486011 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "8f38c63c8c924bd594592f73a7459c1c"
format_stamp: "Formatted at 2026-08-08 02:26:41 on dist-test-slave-zkpd"
server_key: "8e611429b17204d63aa4c66ef27c3d2d"
server_key_iv: "cebb5e8b901e1b87cfbaeba969d8d064"
server_key_version: "encryptionkey@0"
I20260808 02:26:41.487248 21117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:41.487983 21134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:41.488152 21117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:41.488238 21117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "8f38c63c8c924bd594592f73a7459c1c"
format_stamp: "Formatted at 2026-08-08 02:26:41 on dist-test-slave-zkpd"
server_key: "8e611429b17204d63aa4c66ef27c3d2d"
server_key_iv: "cebb5e8b901e1b87cfbaeba969d8d064"
server_key_version: "encryptionkey@0"
I20260808 02:26:41.488294 21117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:41.510077 21117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:41.511741 21117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:41.511853 21117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:41.515844 21117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36315
I20260808 02:26:41.515940 21190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36315 every 8 connection(s)
I20260808 02:26:41.516288 21117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:41.517158 21191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:41.519357 21191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c: Bootstrap starting.
I20260808 02:26:41.519958 21191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:41.520249 21191 log.cc:824] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:41.520953 21191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c: No bootstrap required, opened a new log
I20260808 02:26:41.522504 21191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38c63c8c924bd594592f73a7459c1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260808 02:26:41.522639 21191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:41.522684 21191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38c63c8c924bd594592f73a7459c1c, State: Initialized, Role: FOLLOWER
I20260808 02:26:41.522840 21191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38c63c8c924bd594592f73a7459c1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260808 02:26:41.522929 21191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:41.522981 21191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:41.523034 21191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:41.523564 21191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38c63c8c924bd594592f73a7459c1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260808 02:26:41.523674 21191 leader_election.cc:302] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f38c63c8c924bd594592f73a7459c1c; no voters:
I20260808 02:26:41.523857 21191 leader_election.cc:288] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:41.523887 21194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:41.524111 21194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [term 1 LEADER]: Becoming Leader. State: Replica: 8f38c63c8c924bd594592f73a7459c1c, State: Running, Role: LEADER
I20260808 02:26:41.524155 21191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:41.524230 21194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38c63c8c924bd594592f73a7459c1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260808 02:26:41.524621 21196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f38c63c8c924bd594592f73a7459c1c. Latest consensus state: current_term: 1 leader_uuid: "8f38c63c8c924bd594592f73a7459c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38c63c8c924bd594592f73a7459c1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } } }
I20260808 02:26:41.524765 21196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:41.524755 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f38c63c8c924bd594592f73a7459c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f38c63c8c924bd594592f73a7459c1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } } }
I20260808 02:26:41.525197 21204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:41.525218 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:41.525655 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21117
I20260808 02:26:41.525730 21204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:41.525781 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:41.525978 31360 external_mini_cluster.cc:1428] Setting key a44b3e039b582efc108eec44d8561707
I20260808 02:26:41.528723 21204 catalog_manager.cc:1347] Generated new cluster ID: 309d9f347d274efc8523121133dc752f
I20260808 02:26:41.528776 21204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:41 dist-test-slave-zkpd krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:26:41.539000 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:41.526690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:59399 (local address 127.0.0.1:36315)
0808 02:26:41.528053 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:41.528060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:41.528092 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:26:41.528181 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:41.528187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:41.528230 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:41.528339 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:41.529121 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:41.530508 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:41.531585 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:41.532239 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:41.532336 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:41.535278 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:41.535282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:41.535282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:41.535295 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:41.537246 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:41.537713 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:41.537716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:41.537716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:41.537831 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:41.538068 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:41.538069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:41.538069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:41.538355 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:41.538478 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:41.538662 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 02:26:41.538764 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1328,"thread_start_us":110,"threads_started":1}
I20260808 02:26:41.540325 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:41.544587 21204 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:41.545212 21204 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:41.549139 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:41.541817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:60967 (local address 127.0.0.1:36315)
0808 02:26:41.541849 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:41.541850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:41.541961 (+ 111us) server_negotiation.cc:407] Connection header received
0808 02:26:41.542016 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:41.542017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:41.542026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:41.542079 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:41.542690 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:41.543424 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:41.544510 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:41.544840 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:41.544909 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:41.546272 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:41.546273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:41.546274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:41.546280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:41.547960 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:41.548352 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:41.548353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:41.548354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:41.548420 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:41.548639 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:41.548640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:41.548640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:41.548765 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:41.548886 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:41.548984 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:26:41.549047 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 02:26:41.551959 21204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f38c63c8c924bd594592f73a7459c1c: Generated new TSK 0
I20260808 02:26:41.552206 21204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:41.560422 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21117
W20260808 02:26:41.566710 21102 connection.cc:535] client connection to 127.0.0.1:36315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36315): Connection reset by peer (error 104)
2026-08-08T02:26:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:41 dist-test-slave-zkpd krb5kdc[21223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:41 dist-test-slave-zkpd krb5kdc[21223](info): set up 2 sockets
Aug 08 02:26:41 dist-test-slave-zkpd krb5kdc[21223](info): commencing operation
krb5kdc: starting...
W20260808 02:26:42.611788 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:42 dist-test-slave-zkpd krb5kdc[21223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:42.730325 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:42.812551 21238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:42.812749 21238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:42.812772 21238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:42.814006 21238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:42.814061 21238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:42.814077 21238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:42.814090 21238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:42.814102 21238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:42.815785 21238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:42.816067 21238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:42.816301 21238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:42.821185 21244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:42.821208 21243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:42.821223 21238 server_base.cc:1034] running on GCE node
W20260808 02:26:42.821204 21246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:42.821593 21238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:42.821880 21238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:42.823020 21238 hybrid_clock.cc:648] HybridClock initialized: now 1786156002822997 us; error 37 us; skew 500 ppm
Aug 08 02:26:42 dist-test-slave-zkpd krb5kdc[21223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:42.825563 21238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:42.825999 21238 webserver.cc:460] Webserver started at http://127.0.0.1:45985/ using document root <none> and password file <none>
I20260808 02:26:42.826203 21238 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:42.826246 21238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:42.826407 21238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:42.827526 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "44eed378f6bd43fba8c4f9b507e101cd"
format_stamp: "Formatted at 2026-08-08 02:26:42 on dist-test-slave-zkpd"
server_key: "39e1c802b46eb00710a28e87c823c6f6"
server_key_iv: "f3ea9b3ae15123611d8eec53bcd60168"
server_key_version: "encryptionkey@0"
I20260808 02:26:42.827981 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "44eed378f6bd43fba8c4f9b507e101cd"
format_stamp: "Formatted at 2026-08-08 02:26:42 on dist-test-slave-zkpd"
server_key: "39e1c802b46eb00710a28e87c823c6f6"
server_key_iv: "f3ea9b3ae15123611d8eec53bcd60168"
server_key_version: "encryptionkey@0"
I20260808 02:26:42.829279 21238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:42.829995 21253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:42.830155 21238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:42.830237 21238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "44eed378f6bd43fba8c4f9b507e101cd"
format_stamp: "Formatted at 2026-08-08 02:26:42 on dist-test-slave-zkpd"
server_key: "39e1c802b46eb00710a28e87c823c6f6"
server_key_iv: "f3ea9b3ae15123611d8eec53bcd60168"
server_key_version: "encryptionkey@0"
I20260808 02:26:42.830289 21238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:42.840138 21238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:42.841770 21238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:42.841873 21238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:42.845523 21238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38387
I20260808 02:26:42.845592 21307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38387 every 8 connection(s)
I20260808 02:26:42.845889 21238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:42.846781 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:42.849045 21308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd: Bootstrap starting.
I20260808 02:26:42.849603 21308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:42.849884 21308 log.cc:824] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:42.850538 21308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd: No bootstrap required, opened a new log
I20260808 02:26:42.852164 21308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eed378f6bd43fba8c4f9b507e101cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260808 02:26:42.852298 21308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:42.852350 21308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44eed378f6bd43fba8c4f9b507e101cd, State: Initialized, Role: FOLLOWER
I20260808 02:26:42.852528 21308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eed378f6bd43fba8c4f9b507e101cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260808 02:26:42.852607 21308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:42.852655 21308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:42.852708 21308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:42.853232 21308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eed378f6bd43fba8c4f9b507e101cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260808 02:26:42.853341 21308 leader_election.cc:302] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44eed378f6bd43fba8c4f9b507e101cd; no voters:
I20260808 02:26:42.853510 21308 leader_election.cc:288] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:42.853533 21312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:42.853668 21312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [term 1 LEADER]: Becoming Leader. State: Replica: 44eed378f6bd43fba8c4f9b507e101cd, State: Running, Role: LEADER
I20260808 02:26:42.853799 21308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:42.853824 21312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eed378f6bd43fba8c4f9b507e101cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } }
I20260808 02:26:42.854234 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44eed378f6bd43fba8c4f9b507e101cd. Latest consensus state: current_term: 1 leader_uuid: "44eed378f6bd43fba8c4f9b507e101cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eed378f6bd43fba8c4f9b507e101cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } } }
I20260808 02:26:42.854355 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44eed378f6bd43fba8c4f9b507e101cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eed378f6bd43fba8c4f9b507e101cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38387 } } }
I20260808 02:26:42.854439 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:42.854378 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:42.854861 21321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:42.855146 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21238
I20260808 02:26:42.855278 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:42.855444 21321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:42.855489 31360 external_mini_cluster.cc:1428] Setting key 13cbe2289e449a2d3a88a4ade209ecdc
I20260808 02:26:42.857105 21321 catalog_manager.cc:1347] Generated new cluster ID: c73668e131494cd6b486bec98c497bf5
I20260808 02:26:42.857161 21321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:42.861853 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:42.856052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:55869 (local address 127.0.0.1:38387)
0808 02:26:42.856447 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:42.856472 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:42.856505 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:26:42.856628 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:42.856634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:42.856686 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:42.856818 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:42.857033 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:42.858430 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:42.859596 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:42.860239 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:42.860334 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:42.861076 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:42.861079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:42.861079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:42.861266 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:42.861302 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:42.861314 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:42.861391 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:42.861507 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 02:26:42.861530 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":82,"threads_started":1}
I20260808 02:26:42.862787 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:42.863819 21321 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:42.864524 21321 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:42.867924 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:42.864300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58263 (local address 127.0.0.1:38387)
0808 02:26:42.864338 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:42.864340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:42.864556 (+ 216us) server_negotiation.cc:407] Connection header received
0808 02:26:42.864602 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:42.864604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:42.864617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:42.864686 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:42.864808 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:42.865548 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:42.866627 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:42.866923 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:42.866978 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:42.867643 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:42.867644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:42.867644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:42.867648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:42.867661 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:42.867665 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:42.867719 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:42.867808 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:26:42.867819 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:26:42.871429 21321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44eed378f6bd43fba8c4f9b507e101cd: Generated new TSK 0
I20260808 02:26:42.871603 21321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:42.875320 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21238
W20260808 02:26:42.881994 21221 connection.cc:535] client connection to 127.0.0.1:38387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38387): Connection reset by peer (error 104)
2026-08-08T02:26:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:42 dist-test-slave-zkpd krb5kdc[21341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:42 dist-test-slave-zkpd krb5kdc[21341](info): set up 2 sockets
Aug 08 02:26:42 dist-test-slave-zkpd krb5kdc[21341](info): commencing operation
krb5kdc: starting...
W20260808 02:26:43.935585 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:44 dist-test-slave-zkpd krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:26:44.056941 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:44.137681 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:44.137898 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:44.137945 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:44.139086 21354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:44.139146 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:44.139173 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:44.139204 21354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:44.139226 21354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:44.140779 21354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:44.141050 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:44.141273 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:44.146059 21362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:44.146034 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:44.146045 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:44.146342 21354 server_base.cc:1034] running on GCE node
I20260808 02:26:44.146595 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:44.146899 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:44.148056 21354 hybrid_clock.cc:648] HybridClock initialized: now 1786156004148042 us; error 26 us; skew 500 ppm
Aug 08 02:26:44 dist-test-slave-zkpd krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:44.150547 21354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:26:44.151010 21354 webserver.cc:460] Webserver started at http://127.0.0.1:34251/ using document root <none> and password file <none>
I20260808 02:26:44.151242 21354 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:44.151307 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:44.151458 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:44.152556 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "8e7b14f224bf4bc0a7e47a45cafbe479"
format_stamp: "Formatted at 2026-08-08 02:26:44 on dist-test-slave-zkpd"
server_key: "75ef06e3578f4412f884505950f1e511"
server_key_iv: "d41f68a364e6944de0c9bfde7f2e9b92"
server_key_version: "encryptionkey@0"
I20260808 02:26:44.152922 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "8e7b14f224bf4bc0a7e47a45cafbe479"
format_stamp: "Formatted at 2026-08-08 02:26:44 on dist-test-slave-zkpd"
server_key: "75ef06e3578f4412f884505950f1e511"
server_key_iv: "d41f68a364e6944de0c9bfde7f2e9b92"
server_key_version: "encryptionkey@0"
I20260808 02:26:44.154249 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:44.154937 21369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:44.155102 21354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:44.155200 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "8e7b14f224bf4bc0a7e47a45cafbe479"
format_stamp: "Formatted at 2026-08-08 02:26:44 on dist-test-slave-zkpd"
server_key: "75ef06e3578f4412f884505950f1e511"
server_key_iv: "d41f68a364e6944de0c9bfde7f2e9b92"
server_key_version: "encryptionkey@0"
I20260808 02:26:44.155279 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:44.158530 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:44.160856 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:44.161005 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:44.164973 21354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39955
I20260808 02:26:44.165006 21427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39955 every 8 connection(s)
I20260808 02:26:44.165325 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:44.166275 21428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:44.168541 21428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479: Bootstrap starting.
I20260808 02:26:44.169101 21428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:44.169377 21428 log.cc:824] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:44.170071 21428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479: No bootstrap required, opened a new log
I20260808 02:26:44.171159 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21354
I20260808 02:26:44.171295 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:44.171494 31360 external_mini_cluster.cc:1428] Setting key 5fc52cc97da56e38d2ae7a737adbcf3b
I20260808 02:26:44.172149 21428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260808 02:26:44.172313 21428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:44.172338 21428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7b14f224bf4bc0a7e47a45cafbe479, State: Initialized, Role: FOLLOWER
I20260808 02:26:44.172495 21428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260808 02:26:44.172581 21428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:44.172662 21428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:44.172720 21428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:44.173666 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:44.172012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58977 (local address 127.0.0.1:39955)
0808 02:26:44.172195 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:44.172203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:44.172228 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:26:44.172349 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:44.172356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:44.172388 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:44.172564 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:44.173194 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:44.173197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:44.173198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:44.173201 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:44.173240 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:44.173251 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:44.173304 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:44.173402 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:26:44.173439 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260808 02:26:44.173835 21428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260808 02:26:44.173960 21428 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7b14f224bf4bc0a7e47a45cafbe479; no voters:
I20260808 02:26:44.174115 21428 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:44.174146 21433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:44.174302 21433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7b14f224bf4bc0a7e47a45cafbe479, State: Running, Role: LEADER
I20260808 02:26:44.174386 21433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } }
I20260808 02:26:44.174424 21428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:44.174491 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:44.174714 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } } }
I20260808 02:26:44.174853 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:44.175038 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7b14f224bf4bc0a7e47a45cafbe479. Latest consensus state: current_term: 1 leader_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7b14f224bf4bc0a7e47a45cafbe479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39955 } } }
I20260808 02:26:44.175158 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:44.175536 21440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:44.176113 21440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:44.177193 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:44.176058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:53353 (local address 127.0.0.1:39955)
0808 02:26:44.176108 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:44.176110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:44.176237 (+ 127us) server_negotiation.cc:407] Connection header received
0808 02:26:44.176364 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:44.176365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:44.176369 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:44.176400 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:44.176975 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:44.176976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:44.176977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:44.176980 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:44.176994 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:44.176998 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:44.177060 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:44.177114 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 02:26:44.177127 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:26:44.182902 21440 catalog_manager.cc:1347] Generated new cluster ID: e9fefcf367f94e7ebb27fe06e0340818
I20260808 02:26:44.182984 21440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:44.196997 21440 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:44.197162 21440 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:44.203816 21440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7b14f224bf4bc0a7e47a45cafbe479: Generated new TSK 0
I20260808 02:26:44.204021 21440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:44.210625 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21354
2026-08-08T02:26:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:44 dist-test-slave-zkpd krb5kdc[21460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:44 dist-test-slave-zkpd krb5kdc[21460](info): set up 2 sockets
Aug 08 02:26:44 dist-test-slave-zkpd krb5kdc[21460](info): commencing operation
krb5kdc: starting...
W20260808 02:26:45.276201 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:45 dist-test-slave-zkpd krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:45.392963 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34673
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:45.474134 21473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:45.474328 21473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:45.474351 21473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:45.475440 21473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:45.475472 21473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:45.475486 21473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:45.475499 21473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:45.475512 21473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:45.476979 21473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:34673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:45.477167 21473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:45.477330 21473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:26:45.482025 21473 server_base.cc:1034] running on GCE node
W20260808 02:26:45.481982 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:45.481962 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:45.481984 21481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:45.482474 21473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:45.482753 21473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:45.483937 21473 hybrid_clock.cc:648] HybridClock initialized: now 1786156005483886 us; error 68 us; skew 500 ppm
Aug 08 02:26:45 dist-test-slave-zkpd krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156005, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:45.486315 21473 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:45.486833 21473 webserver.cc:460] Webserver started at http://127.30.160.62:39247/ using document root <none> and password file <none>
I20260808 02:26:45.487095 21473 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:45.487175 21473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:45.487316 21473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:45.488576 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "9c52d6effd9049fdbc8797727644eb7d"
format_stamp: "Formatted at 2026-08-08 02:26:45 on dist-test-slave-zkpd"
server_key: "786ecb12c3ec7ba26559db410dc342ce"
server_key_iv: "e502cca2996427fd38a919bda44a09e7"
server_key_version: "encryptionkey@0"
I20260808 02:26:45.488936 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "9c52d6effd9049fdbc8797727644eb7d"
format_stamp: "Formatted at 2026-08-08 02:26:45 on dist-test-slave-zkpd"
server_key: "786ecb12c3ec7ba26559db410dc342ce"
server_key_iv: "e502cca2996427fd38a919bda44a09e7"
server_key_version: "encryptionkey@0"
I20260808 02:26:45.490257 21473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:26:45.491045 21488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:45.491240 21473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:45.491333 21473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "9c52d6effd9049fdbc8797727644eb7d"
format_stamp: "Formatted at 2026-08-08 02:26:45 on dist-test-slave-zkpd"
server_key: "786ecb12c3ec7ba26559db410dc342ce"
server_key_iv: "e502cca2996427fd38a919bda44a09e7"
server_key_version: "encryptionkey@0"
I20260808 02:26:45.491430 21473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:45.505091 21473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:45.506760 21473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:45.506906 21473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:45.510841 21473 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:34673
I20260808 02:26:45.510874 21542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:34673 every 8 connection(s)
I20260808 02:26:45.511201 21473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:45.512064 21543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:45.514209 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d: Bootstrap starting.
I20260808 02:26:45.514724 21543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:45.514966 21543 log.cc:824] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:45.515573 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d: No bootstrap required, opened a new log
I20260808 02:26:45.517134 21543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c52d6effd9049fdbc8797727644eb7d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34673 } }
I20260808 02:26:45.517267 21543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:45.517259 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21473
I20260808 02:26:45.517289 21543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c52d6effd9049fdbc8797727644eb7d, State: Initialized, Role: FOLLOWER
I20260808 02:26:45.517416 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:45.517446 21543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c52d6effd9049fdbc8797727644eb7d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34673 } }
I20260808 02:26:45.517550 21543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:45.517592 21543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:45.517643 21543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:45.517652 31360 external_mini_cluster.cc:1428] Setting key 5244e138e9c651884f73f16b27e968e4
I20260808 02:26:45.518687 21543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c52d6effd9049fdbc8797727644eb7d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34673 } }
I20260808 02:26:45.518816 21543 leader_election.cc:302] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c52d6effd9049fdbc8797727644eb7d; no voters:
I20260808 02:26:45.518994 21543 leader_election.cc:288] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:45.519016 21548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:45.519205 21548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [term 1 LEADER]: Becoming Leader. State: Replica: 9c52d6effd9049fdbc8797727644eb7d, State: Running, Role: LEADER
I20260808 02:26:45.519250 21543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:45.519332 21548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c52d6effd9049fdbc8797727644eb7d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34673 } }
I20260808 02:26:45.519734 21548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c52d6effd9049fdbc8797727644eb7d. Latest consensus state: current_term: 1 leader_uuid: "9c52d6effd9049fdbc8797727644eb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c52d6effd9049fdbc8797727644eb7d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34673 } } }
I20260808 02:26:45.519852 21548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:45.520457 21557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:45.521055 21557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:45.521121 21550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c52d6effd9049fdbc8797727644eb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c52d6effd9049fdbc8797727644eb7d" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34673 } } }
I20260808 02:26:45.521199 21550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:45.522403 21557 catalog_manager.cc:1347] Generated new cluster ID: 2411b45ac8304af59b9997f17549d54e
I20260808 02:26:45.522466 21557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:26:45 dist-test-slave-zkpd krb5kdc[21460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:45.529276 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:45.518111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49101 (local address 127.30.160.62:34673)
0808 02:26:45.518291 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:45.518300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:45.518328 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:26:45.518450 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:45.518455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:45.518494 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:45.518597 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:45.519415 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:45.520858 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:45.521920 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:45.522479 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:45.522587 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:45.525524 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:45.525528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:45.525529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:45.525540 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:45.527447 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:45.528000 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:45.528003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:45.528003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:45.528117 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:45.528371 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:45.528372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:45.528372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:45.528654 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:45.528798 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:45.528958 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:26:45.529052 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260808 02:26:45.530453 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:45.532866 21557 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:45.533340 21557 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:45.539599 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:45.531989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43121 (local address 127.30.160.62:34673)
0808 02:26:45.532024 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:45.532026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:45.532622 (+ 596us) server_negotiation.cc:407] Connection header received
0808 02:26:45.532693 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:45.532694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:45.532705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:45.532752 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:45.533403 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:45.534109 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:45.535141 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:45.535439 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:45.535493 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:45.536875 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:45.536876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:45.536876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:45.536883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:45.538395 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:45.538800 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:45.538801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:45.538801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:45.538854 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:45.539074 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:45.539075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:45.539075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:45.539184 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:45.539267 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:45.539453 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 02:26:45.539513 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:45.540518 21557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c52d6effd9049fdbc8797727644eb7d: Generated new TSK 0
I20260808 02:26:45.540694 21557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:45.543797 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21473
W20260808 02:26:45.549782 21458 connection.cc:535] client connection to 127.30.160.62:34673 recv error: Network error: failed to read from TLS socket (remote: 127.30.160.62:34673): Connection reset by peer (error 104)
2026-08-08T02:26:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:45 dist-test-slave-zkpd krb5kdc[21577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:45 dist-test-slave-zkpd krb5kdc[21577](info): set up 2 sockets
Aug 08 02:26:45 dist-test-slave-zkpd krb5kdc[21577](info): commencing operation
krb5kdc: starting...
W20260808 02:26:46.607721 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:46 dist-test-slave-zkpd krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:46.726269 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44635
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:46.806728 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:46.806934 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:46.806957 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:46.808102 21590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:46.808140 21590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:46.808154 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:46.808204 21590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:46.808224 21590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:46.809655 21590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:44635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:46.809892 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:46.810107 21590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:46.814829 21597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:46.814940 21590 server_base.cc:1034] running on GCE node
W20260808 02:26:46.814858 21596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:46.815037 21599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:46.815249 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:46.815569 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:46.816726 21590 hybrid_clock.cc:648] HybridClock initialized: now 1786156006816714 us; error 26 us; skew 500 ppm
Aug 08 02:26:46 dist-test-slave-zkpd krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:46.823253 21590 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:46.823679 21590 webserver.cc:460] Webserver started at http://127.30.160.62:39375/ using document root <none> and password file <none>
I20260808 02:26:46.823935 21590 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:46.824007 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:46.824154 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:46.825188 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "e059f685606c4f73b10fb3237c13d37c"
format_stamp: "Formatted at 2026-08-08 02:26:46 on dist-test-slave-zkpd"
server_key: "411f58754a0ec5bef3a019cc5acd6ffe"
server_key_iv: "0214d54ebdf12c0ecb83c6008e4e41c4"
server_key_version: "encryptionkey@0"
I20260808 02:26:46.825552 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "e059f685606c4f73b10fb3237c13d37c"
format_stamp: "Formatted at 2026-08-08 02:26:46 on dist-test-slave-zkpd"
server_key: "411f58754a0ec5bef3a019cc5acd6ffe"
server_key_iv: "0214d54ebdf12c0ecb83c6008e4e41c4"
server_key_version: "encryptionkey@0"
I20260808 02:26:46.826884 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:46.827576 21606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:46.827739 21590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:46.827837 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "e059f685606c4f73b10fb3237c13d37c"
format_stamp: "Formatted at 2026-08-08 02:26:46 on dist-test-slave-zkpd"
server_key: "411f58754a0ec5bef3a019cc5acd6ffe"
server_key_iv: "0214d54ebdf12c0ecb83c6008e4e41c4"
server_key_version: "encryptionkey@0"
I20260808 02:26:46.827944 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:46.853762 21590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:46.855595 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:46.855757 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:46.859596 21590 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:44635
I20260808 02:26:46.859633 21659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:44635 every 8 connection(s)
I20260808 02:26:46.859997 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:46.860884 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:46.861552 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21590
I20260808 02:26:46.861693 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:46.861878 31360 external_mini_cluster.cc:1428] Setting key 6b35725f6024ef94d98a33e670e745d4
I20260808 02:26:46.863946 21660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c: Bootstrap starting.
I20260808 02:26:46.864586 21660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:46.864908 21660 log.cc:824] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:46.865619 21660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c: No bootstrap required, opened a new log
I20260808 02:26:46.867308 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:46.862370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45203 (local address 127.30.160.62:44635)
0808 02:26:46.862541 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:46.862548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:46.862573 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:26:46.862962 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:46.862968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:46.863012 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:46.863124 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:46.863394 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:46.864383 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:46.865420 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:46.865847 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:46.865930 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:46.866643 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:46.866647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:46.866647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:46.866802 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:46.866830 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:46.866840 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:46.866904 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:46.867036 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 02:26:46.867080 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260808 02:26:46.868105 21660 raft_consensus.cc:348] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e059f685606c4f73b10fb3237c13d37c" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44635 } }
I20260808 02:26:46.868275 21660 raft_consensus.cc:374] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:46.868353 21660 raft_consensus.cc:729] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e059f685606c4f73b10fb3237c13d37c, State: Initialized, Role: FOLLOWER
I20260808 02:26:46.868381 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:46.868489 21660 consensus_queue.cc:260] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e059f685606c4f73b10fb3237c13d37c" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44635 } }
I20260808 02:26:46.868564 21660 raft_consensus.cc:388] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:46.868599 21660 raft_consensus.cc:482] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:46.868646 21660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:46.869190 21660 raft_consensus.cc:504] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e059f685606c4f73b10fb3237c13d37c" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44635 } }
I20260808 02:26:46.869303 21660 leader_election.cc:302] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e059f685606c4f73b10fb3237c13d37c; no voters:
I20260808 02:26:46.869458 21660 leader_election.cc:288] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:46.869488 21669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:46.869601 21669 raft_consensus.cc:686] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [term 1 LEADER]: Becoming Leader. State: Replica: e059f685606c4f73b10fb3237c13d37c, State: Running, Role: LEADER
I20260808 02:26:46.869709 21660 sys_catalog.cc:564] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:46.869728 21669 consensus_queue.cc:237] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e059f685606c4f73b10fb3237c13d37c" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44635 } }
I20260808 02:26:46.870428 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e059f685606c4f73b10fb3237c13d37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e059f685606c4f73b10fb3237c13d37c" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44635 } } }
I20260808 02:26:46.870414 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e059f685606c4f73b10fb3237c13d37c. Latest consensus state: current_term: 1 leader_uuid: "e059f685606c4f73b10fb3237c13d37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e059f685606c4f73b10fb3237c13d37c" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 44635 } } }
I20260808 02:26:46.870514 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:46.870522 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:46.870764 21679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:46.871991 21679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:46.873775 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:46.869833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48989 (local address 127.30.160.62:44635)
0808 02:26:46.869866 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:46.869868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:46.869935 (+ 67us) server_negotiation.cc:407] Connection header received
0808 02:26:46.870097 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:46.870098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:46.870107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:46.870155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:46.870347 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:46.871330 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:46.872490 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:46.872771 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:46.872841 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:46.873479 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:46.873480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:46.873481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:46.873485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:46.873499 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:46.873507 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:46.873551 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:46.873681 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 02:26:46.873692 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:46.874104 21679 catalog_manager.cc:1347] Generated new cluster ID: 7af2fb99962f4b02aceeefca9464ae16
I20260808 02:26:46.874174 21679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:46.884371 21679 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:46.884886 21679 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:46.888234 21679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e059f685606c4f73b10fb3237c13d37c: Generated new TSK 0
I20260808 02:26:46.888393 21679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:46.890492 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21590
W20260808 02:26:46.896005 21575 connection.cc:535] client connection to 127.30.160.62:44635 recv error: Network error: failed to read from TLS socket (remote: 127.30.160.62:44635): Connection reset by peer (error 104)
2026-08-08T02:26:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:46 dist-test-slave-zkpd krb5kdc[21691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:46 dist-test-slave-zkpd krb5kdc[21691](info): set up 2 sockets
Aug 08 02:26:46 dist-test-slave-zkpd krb5kdc[21691](info): commencing operation
krb5kdc: starting...
W20260808 02:26:47.959988 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:48 dist-test-slave-zkpd krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:48.082521 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37435
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:37435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:48.164312 21705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:48.164515 21705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:48.164538 21705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:48.165665 21705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:48.165706 21705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:48.165720 21705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:48.165762 21705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:48.165781 21705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:48.167264 21705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:37435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:37435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:48.167508 21705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:48.167724 21705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:48.172433 21710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:48.172461 21711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:48.172433 21713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:48.172667 21705 server_base.cc:1034] running on GCE node
I20260808 02:26:48.172997 21705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:48.173274 21705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:48.174439 21705 hybrid_clock.cc:648] HybridClock initialized: now 1786156008174426 us; error 40 us; skew 500 ppm
Aug 08 02:26:48 dist-test-slave-zkpd krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156008, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:48.176923 21705 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:48.177382 21705 webserver.cc:460] Webserver started at http://127.30.160.62:33273/ using document root <none> and password file <none>
I20260808 02:26:48.177591 21705 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:48.177659 21705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:48.177798 21705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:48.178813 21705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "6ea45f6823614cc79fe6758052f8992f"
format_stamp: "Formatted at 2026-08-08 02:26:48 on dist-test-slave-zkpd"
server_key: "5ff4504decfb110619c2c7d70baea152"
server_key_iv: "0f26efb85ad0fcb483de6a628ee8ad0e"
server_key_version: "encryptionkey@0"
I20260808 02:26:48.179180 21705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "6ea45f6823614cc79fe6758052f8992f"
format_stamp: "Formatted at 2026-08-08 02:26:48 on dist-test-slave-zkpd"
server_key: "5ff4504decfb110619c2c7d70baea152"
server_key_iv: "0f26efb85ad0fcb483de6a628ee8ad0e"
server_key_version: "encryptionkey@0"
I20260808 02:26:48.180559 21705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:48.181255 21722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:48.181519 21705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:48.181619 21705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "6ea45f6823614cc79fe6758052f8992f"
format_stamp: "Formatted at 2026-08-08 02:26:48 on dist-test-slave-zkpd"
server_key: "5ff4504decfb110619c2c7d70baea152"
server_key_iv: "0f26efb85ad0fcb483de6a628ee8ad0e"
server_key_version: "encryptionkey@0"
I20260808 02:26:48.181689 21705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:48.184943 21705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:48.187114 21705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:48.187259 21705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:48.191239 21705 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:37435
I20260808 02:26:48.191272 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:37435 every 8 connection(s)
I20260808 02:26:48.191599 21705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:48.192574 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:48.194748 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f: Bootstrap starting.
I20260808 02:26:48.195308 21777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:48.195588 21777 log.cc:824] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:48.196302 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f: No bootstrap required, opened a new log
I20260808 02:26:48.197336 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21705
I20260808 02:26:48.197464 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:48.197620 31360 external_mini_cluster.cc:1428] Setting key 75de7a67c6d13b2c33e8edfd21848b78
I20260808 02:26:48.198164 21777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea45f6823614cc79fe6758052f8992f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37435 } }
I20260808 02:26:48.198361 21777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:48.198402 21777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea45f6823614cc79fe6758052f8992f, State: Initialized, Role: FOLLOWER
I20260808 02:26:48.198522 21777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea45f6823614cc79fe6758052f8992f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37435 } }
I20260808 02:26:48.198601 21777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:48.198637 21777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:48.198727 21777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:48.199726 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:48.198124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41417 (local address 127.30.160.62:37435)
0808 02:26:48.198329 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:48.198338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:48.198373 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:26:48.198483 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:48.198491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:48.198519 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:48.198665 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:48.199295 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:48.199300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:48.199301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:48.199308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:48.199355 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:48.199368 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:48.199450 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:48.199489 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:26:48.199528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260808 02:26:48.199826 21777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea45f6823614cc79fe6758052f8992f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37435 } }
I20260808 02:26:48.199945 21777 leader_election.cc:302] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ea45f6823614cc79fe6758052f8992f; no voters:
I20260808 02:26:48.200105 21777 leader_election.cc:288] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:48.200132 21782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:48.200374 21777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:48.200423 21782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [term 1 LEADER]: Becoming Leader. State: Replica: 6ea45f6823614cc79fe6758052f8992f, State: Running, Role: LEADER
I20260808 02:26:48.200578 21782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea45f6823614cc79fe6758052f8992f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37435 } }
I20260808 02:26:48.200943 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:48.201138 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ea45f6823614cc79fe6758052f8992f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea45f6823614cc79fe6758052f8992f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37435 } } }
I20260808 02:26:48.201216 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:48.201382 21784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ea45f6823614cc79fe6758052f8992f. Latest consensus state: current_term: 1 leader_uuid: "6ea45f6823614cc79fe6758052f8992f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea45f6823614cc79fe6758052f8992f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 37435 } } }
I20260808 02:26:48.201449 21784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:48.201555 21791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:48.202064 21791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:48.204246 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:48.202945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36759 (local address 127.30.160.62:37435)
0808 02:26:48.202981 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:48.202984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:48.203269 (+ 285us) server_negotiation.cc:407] Connection header received
0808 02:26:48.203341 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:48.203342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:48.203348 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:48.203404 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:48.204017 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:48.204018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:48.204018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:48.204023 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:48.204036 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:48.204040 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:48.204096 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:48.204151 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 02:26:48.204166 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:48.209230 21791 catalog_manager.cc:1347] Generated new cluster ID: 3f4dc8f2d9fd45559e160e10c2c5edfa
I20260808 02:26:48.209281 21791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:48.229338 21791 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:48.229555 21791 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:48.233917 21791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ea45f6823614cc79fe6758052f8992f: Generated new TSK 0
I20260808 02:26:48.234114 21791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:48.242131 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21705
2026-08-08T02:26:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:48 dist-test-slave-zkpd krb5kdc[21810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:48 dist-test-slave-zkpd krb5kdc[21810](info): set up 2 sockets
Aug 08 02:26:48 dist-test-slave-zkpd krb5kdc[21810](info): commencing operation
krb5kdc: starting...
W20260808 02:26:49.288344 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:49 dist-test-slave-zkpd krb5kdc[21810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:49.403659 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39283
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:39283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:49.487156 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:49.487355 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:49.487377 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:49.488662 21823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:49.488708 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:49.488721 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:49.488734 21823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:49.488777 21823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:49.490231 21823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:39283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:39283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:49.490471 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:49.490696 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:49.495424 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:49.495461 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:49.495625 21823 server_base.cc:1034] running on GCE node
W20260808 02:26:49.495433 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:49.495960 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:49.496256 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:49.497416 21823 hybrid_clock.cc:648] HybridClock initialized: now 1786156009497402 us; error 35 us; skew 500 ppm
Aug 08 02:26:49 dist-test-slave-zkpd krb5kdc[21810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156009, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:49.500444 21823 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:49.500888 21823 webserver.cc:460] Webserver started at http://127.30.160.62:43657/ using document root <none> and password file <none>
I20260808 02:26:49.501113 21823 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:49.501178 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:49.501323 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:49.502341 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "4d51d215c0ce4186ac3d3d23b1693f98"
format_stamp: "Formatted at 2026-08-08 02:26:49 on dist-test-slave-zkpd"
server_key: "2e1be129c076c59ed1d8c3d778930f05"
server_key_iv: "82d572e40368233bf6a647e21892fda1"
server_key_version: "encryptionkey@0"
I20260808 02:26:49.502691 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "4d51d215c0ce4186ac3d3d23b1693f98"
format_stamp: "Formatted at 2026-08-08 02:26:49 on dist-test-slave-zkpd"
server_key: "2e1be129c076c59ed1d8c3d778930f05"
server_key_iv: "82d572e40368233bf6a647e21892fda1"
server_key_version: "encryptionkey@0"
I20260808 02:26:49.504004 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:49.504714 21838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:49.504870 21823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:49.504964 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "4d51d215c0ce4186ac3d3d23b1693f98"
format_stamp: "Formatted at 2026-08-08 02:26:49 on dist-test-slave-zkpd"
server_key: "2e1be129c076c59ed1d8c3d778930f05"
server_key_iv: "82d572e40368233bf6a647e21892fda1"
server_key_version: "encryptionkey@0"
I20260808 02:26:49.505044 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:49.541594 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:49.543982 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:49.544145 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:49.548065 21823 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:39283
I20260808 02:26:49.548113 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:39283 every 8 connection(s)
I20260808 02:26:49.548480 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:49.548713 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21823
I20260808 02:26:49.548847 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:49.549034 31360 external_mini_cluster.cc:1428] Setting key 0431cb03ea5cefb4fbf2e9fd52b9252f
I20260808 02:26:49.549690 21893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:49.552204 21893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98: Bootstrap starting.
I20260808 02:26:49.552810 21893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:49.553099 21893 log.cc:824] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:49.553766 21893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98: No bootstrap required, opened a new log
Aug 08 02:26:49 dist-test-slave-zkpd krb5kdc[21810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:49.555605 21893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39283 } }
I20260808 02:26:49.555754 21893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:49.555823 21893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d51d215c0ce4186ac3d3d23b1693f98, State: Initialized, Role: FOLLOWER
I20260808 02:26:49.555995 21893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39283 } }
I20260808 02:26:49.556095 21893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:49.556133 21893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:49.556238 21893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:49.556885 21893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39283 } }
I20260808 02:26:49.556999 21893 leader_election.cc:302] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d51d215c0ce4186ac3d3d23b1693f98; no voters:
I20260808 02:26:49.557195 21893 leader_election.cc:288] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:49.557251 21898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:49.557405 21898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [term 1 LEADER]: Becoming Leader. State: Replica: 4d51d215c0ce4186ac3d3d23b1693f98, State: Running, Role: LEADER
I20260808 02:26:49.557459 21893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:49.557585 21898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39283 } }
I20260808 02:26:49.557936 21900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d51d215c0ce4186ac3d3d23b1693f98. Latest consensus state: current_term: 1 leader_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39283 } } }
I20260808 02:26:49.558070 21900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:49.558336 21907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:49.558822 21907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:49.559813 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:49.549517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56557 (local address 127.30.160.62:39283)
0808 02:26:49.549702 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:49.549716 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:49.549743 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:26:49.550213 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:49.550220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:49.550267 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:49.550408 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:49.551085 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:49.552021 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:49.552833 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:49.553315 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:49.553436 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:49.555698 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:49.555702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:49.555702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:49.555714 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:49.557728 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:49.558237 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:49.558239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:49.558240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:49.558374 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:49.558649 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:49.558650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:49.558650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:49.558996 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:49.559160 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:49.559460 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 02:26:49.559620 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260808 02:26:49.559883 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d51d215c0ce4186ac3d3d23b1693f98" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 39283 } } }
I20260808 02:26:49.559973 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:49.560839 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:49.562100 21907 catalog_manager.cc:1347] Generated new cluster ID: 452423a8813a4b68b101d00e2f8c1b29
I20260808 02:26:49.562158 21907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:49.569250 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:49.562273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55281 (local address 127.30.160.62:39283)
0808 02:26:49.562308 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:49.562310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:49.562443 (+ 133us) server_negotiation.cc:407] Connection header received
0808 02:26:49.562519 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:49.562519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:49.562529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:49.562588 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:49.563152 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:49.563928 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:49.564893 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:49.565199 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:49.565268 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:49.566517 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:49.566519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:49.566519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:49.566526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:49.568136 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:49.568497 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:49.568498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:49.568498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:49.568560 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:49.568752 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:49.568753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:49.568753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:49.568883 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:49.568984 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:49.569090 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:26:49.569164 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 02:26:49.586287 21907 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:49.586796 21907 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:49.597127 21907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d51d215c0ce4186ac3d3d23b1693f98: Generated new TSK 0
I20260808 02:26:49.597455 21907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:49.605556 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21823
2026-08-08T02:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:49 dist-test-slave-zkpd krb5kdc[21925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:49 dist-test-slave-zkpd krb5kdc[21925](info): set up 2 sockets
Aug 08 02:26:49 dist-test-slave-zkpd krb5kdc[21925](info): commencing operation
krb5kdc: starting...
W20260808 02:26:50.664002 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:50 dist-test-slave-zkpd krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:50.782241 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:33197
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:50.862579 21941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:50.862778 21941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:50.862802 21941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:50.863857 21941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:50.863907 21941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:50.863929 21941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:50.863955 21941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:50.864010 21941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:50.865454 21941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:33197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:50.865675 21941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:50.865899 21941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:50.870499 21949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:50.870517 21947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:50.870517 21951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:50.870573 21941 server_base.cc:1034] running on GCE node
I20260808 02:26:50.870988 21941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:50.871300 21941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:50.872453 21941 hybrid_clock.cc:648] HybridClock initialized: now 1786156010872439 us; error 39 us; skew 500 ppm
Aug 08 02:26:50 dist-test-slave-zkpd krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156010, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:50.874769 21941 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:50.875217 21941 webserver.cc:460] Webserver started at http://127.30.160.62:34709/ using document root <none> and password file <none>
I20260808 02:26:50.875445 21941 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:50.875511 21941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:50.875653 21941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:50.876746 21941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "93b38e020dd84c5c95ec4d6d1210693a"
format_stamp: "Formatted at 2026-08-08 02:26:50 on dist-test-slave-zkpd"
server_key: "9879f0920ae7367445af31a8cb47a3d0"
server_key_iv: "f5af191449258902dd705e63917f31af"
server_key_version: "encryptionkey@0"
I20260808 02:26:50.877123 21941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "93b38e020dd84c5c95ec4d6d1210693a"
format_stamp: "Formatted at 2026-08-08 02:26:50 on dist-test-slave-zkpd"
server_key: "9879f0920ae7367445af31a8cb47a3d0"
server_key_iv: "f5af191449258902dd705e63917f31af"
server_key_version: "encryptionkey@0"
I20260808 02:26:50.878437 21941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:26:50.879140 21958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:50.879305 21941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:50.879402 21941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "93b38e020dd84c5c95ec4d6d1210693a"
format_stamp: "Formatted at 2026-08-08 02:26:50 on dist-test-slave-zkpd"
server_key: "9879f0920ae7367445af31a8cb47a3d0"
server_key_iv: "f5af191449258902dd705e63917f31af"
server_key_version: "encryptionkey@0"
I20260808 02:26:50.879482 21941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:50.915342 21941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:50.917207 21941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:50.917369 21941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:50.921469 21941 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:33197
I20260808 02:26:50.921509 22012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:33197 every 8 connection(s)
I20260808 02:26:50.921878 21941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:50.922721 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:50.925061 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a: Bootstrap starting.
I20260808 02:26:50.925639 22014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:50.925910 22014 log.cc:824] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:50.926577 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a: No bootstrap required, opened a new log
I20260808 02:26:50.927168 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 21941
I20260808 02:26:50.927299 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:50.927490 31360 external_mini_cluster.cc:1428] Setting key b253dab820cd1c5e6f851b82e16d89fa
I20260808 02:26:50.929147 22014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b38e020dd84c5c95ec4d6d1210693a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33197 } }
I20260808 02:26:50.929318 22014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:50.929404 22014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b38e020dd84c5c95ec4d6d1210693a, State: Initialized, Role: FOLLOWER
I20260808 02:26:50.929581 22014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b38e020dd84c5c95ec4d6d1210693a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33197 } }
I20260808 02:26:50.929666 22014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:50.929710 22014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:50.929756 22014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:50.930403 22014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b38e020dd84c5c95ec4d6d1210693a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33197 } }
I20260808 02:26:50.930508 22014 leader_election.cc:302] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b38e020dd84c5c95ec4d6d1210693a; no voters:
I20260808 02:26:50.930657 22014 leader_election.cc:288] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:50.930711 22019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:50.930887 22014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:50.930902 22019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [term 1 LEADER]: Becoming Leader. State: Replica: 93b38e020dd84c5c95ec4d6d1210693a, State: Running, Role: LEADER
I20260808 02:26:50.931099 22019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b38e020dd84c5c95ec4d6d1210693a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33197 } }
I20260808 02:26:50.931473 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b38e020dd84c5c95ec4d6d1210693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b38e020dd84c5c95ec4d6d1210693a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33197 } } }
I20260808 02:26:50.931550 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:50.931715 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b38e020dd84c5c95ec4d6d1210693a. Latest consensus state: current_term: 1 leader_uuid: "93b38e020dd84c5c95ec4d6d1210693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b38e020dd84c5c95ec4d6d1210693a" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 33197 } } }
I20260808 02:26:50.931769 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:50.932147 22028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:50.932618 22028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:50.934501 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:50.927975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43557 (local address 127.30.160.62:33197)
0808 02:26:50.928162 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:50.928172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:50.928210 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:26:50.929086 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:50.929094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:50.929141 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:50.929286 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:50.929590 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:50.930981 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:50.932243 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:50.933016 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:50.933149 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:50.933852 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:50.933857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:50.933862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:50.934036 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:50.934067 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:50.934078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:50.934169 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:50.934245 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:26:50.934310 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260808 02:26:50.935475 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:50.936281 22028 catalog_manager.cc:1347] Generated new cluster ID: 96dafb30720c486bad8072f56d2062c4
I20260808 02:26:50.936348 22028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:50.943522 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:50.937024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41509 (local address 127.30.160.62:33197)
0808 02:26:50.937058 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:50.937060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:50.940227 (+ 3167us) server_negotiation.cc:407] Connection header received
0808 02:26:50.940253 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:50.940253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:50.940265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:50.940314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:50.940497 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:50.941184 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:50.942283 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:50.942561 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:50.942628 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:50.943293 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:50.943294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:50.943295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:50.943299 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:50.943312 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:50.943316 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:50.943371 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:50.943432 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 02:26:50.943451 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:50.955211 22028 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:50.955951 22028 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:50.960426 22028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b38e020dd84c5c95ec4d6d1210693a: Generated new TSK 0
I20260808 02:26:50.960592 22028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:50.963268 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 21941
2026-08-08T02:26:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:51 dist-test-slave-zkpd krb5kdc[22045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:51 dist-test-slave-zkpd krb5kdc[22045](info): set up 2 sockets
Aug 08 02:26:51 dist-test-slave-zkpd krb5kdc[22045](info): commencing operation
krb5kdc: starting...
W20260808 02:26:52.025539 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:52 dist-test-slave-zkpd krb5kdc[22045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:52.147207 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45949
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:45949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:52.234668 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:52.234900 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:52.234926 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:52.236225 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:52.236271 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:52.236287 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:52.236331 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:52.236364 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:52.237991 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:45949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:45949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:52.238274 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:52.238535 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:52.243688 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:52.243683 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:52.243748 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:52.244025 22059 server_base.cc:1034] running on GCE node
I20260808 02:26:52.244262 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:52.244578 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:52.245721 22059 hybrid_clock.cc:648] HybridClock initialized: now 1786156012245701 us; error 40 us; skew 500 ppm
Aug 08 02:26:52 dist-test-slave-zkpd krb5kdc[22045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156012, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:52.252882 22059 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:52.253373 22059 webserver.cc:460] Webserver started at http://127.30.160.62:33249/ using document root <none> and password file <none>
I20260808 02:26:52.253614 22059 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:52.253683 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:52.253850 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:52.254900 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "27811fa393224a67b86aedcbdd7db85f"
format_stamp: "Formatted at 2026-08-08 02:26:52 on dist-test-slave-zkpd"
server_key: "cd2d75dce6065a123b8c4efbe3fedd68"
server_key_iv: "0b77f0a02cb1c3add251d1f327aa833a"
server_key_version: "encryptionkey@0"
I20260808 02:26:52.255280 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "27811fa393224a67b86aedcbdd7db85f"
format_stamp: "Formatted at 2026-08-08 02:26:52 on dist-test-slave-zkpd"
server_key: "cd2d75dce6065a123b8c4efbe3fedd68"
server_key_iv: "0b77f0a02cb1c3add251d1f327aa833a"
server_key_version: "encryptionkey@0"
I20260808 02:26:52.256712 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:26:52.257481 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:52.257678 22059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:52.257783 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "27811fa393224a67b86aedcbdd7db85f"
format_stamp: "Formatted at 2026-08-08 02:26:52 on dist-test-slave-zkpd"
server_key: "cd2d75dce6065a123b8c4efbe3fedd68"
server_key_iv: "0b77f0a02cb1c3add251d1f327aa833a"
server_key_version: "encryptionkey@0"
I20260808 02:26:52.257858 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:52.261188 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:52.263446 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:52.263603 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:52.267944 22059 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:45949
I20260808 02:26:52.268042 22130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:45949 every 8 connection(s)
I20260808 02:26:52.268361 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:52.269405 22131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:52.271608 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f: Bootstrap starting.
I20260808 02:26:52.272269 22131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:52.272398 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22059
I20260808 02:26:52.272570 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:52.272598 22131 log.cc:824] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:52.272778 31360 external_mini_cluster.cc:1428] Setting key e7075ff6cc2c703811a664d1c9d4f742
I20260808 02:26:52.273468 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f: No bootstrap required, opened a new log
I20260808 02:26:52.275457 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:52.273305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39701 (local address 127.30.160.62:45949)
0808 02:26:52.273477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:52.273488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:52.273524 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:26:52.273973 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:52.273980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:52.274003 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:52.274150 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:52.274879 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:52.274882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:52.274883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:52.274887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:52.274939 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:52.274949 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:52.275041 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:52.275094 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 02:26:52.275134 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260808 02:26:52.275432 22131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27811fa393224a67b86aedcbdd7db85f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45949 } }
I20260808 02:26:52.275633 22131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:52.275696 22131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27811fa393224a67b86aedcbdd7db85f, State: Initialized, Role: FOLLOWER
I20260808 02:26:52.275825 22131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27811fa393224a67b86aedcbdd7db85f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45949 } }
I20260808 02:26:52.275939 22131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:52.275990 22131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:52.276042 22131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:52.276451 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:52.276714 22131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27811fa393224a67b86aedcbdd7db85f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45949 } }
I20260808 02:26:52.276823 22131 leader_election.cc:302] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27811fa393224a67b86aedcbdd7db85f; no voters:
I20260808 02:26:52.276993 22131 leader_election.cc:288] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:52.277014 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:52.277143 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [term 1 LEADER]: Becoming Leader. State: Replica: 27811fa393224a67b86aedcbdd7db85f, State: Running, Role: LEADER
I20260808 02:26:52.277253 22131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:52.277277 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27811fa393224a67b86aedcbdd7db85f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45949 } }
I20260808 02:26:52.277657 22136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27811fa393224a67b86aedcbdd7db85f. Latest consensus state: current_term: 1 leader_uuid: "27811fa393224a67b86aedcbdd7db85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27811fa393224a67b86aedcbdd7db85f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45949 } } }
I20260808 02:26:52.277745 22136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:52.277669 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27811fa393224a67b86aedcbdd7db85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27811fa393224a67b86aedcbdd7db85f" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 45949 } } }
I20260808 02:26:52.277832 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:52.278034 22146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:52.278965 22146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:52.279454 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:52.278106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53821 (local address 127.30.160.62:45949)
0808 02:26:52.278152 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:52.278154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:52.278221 (+ 67us) server_negotiation.cc:407] Connection header received
0808 02:26:52.278294 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:52.278295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:52.278300 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:52.278352 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:52.279214 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:52.279215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:52.279216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:26:52.279220 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:52.279245 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:26:52.279249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:52.279280 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:52.279386 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:26:52.279398 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:26:52.281160 22146 catalog_manager.cc:1347] Generated new cluster ID: ac6978cda0364ee0bda0ec99d7d6e720
I20260808 02:26:52.281219 22146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:52.300159 22146 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:52.300364 22146 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:52.305303 22146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27811fa393224a67b86aedcbdd7db85f: Generated new TSK 0
I20260808 02:26:52.305500 22146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:52.310635 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22059
2026-08-08T02:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1347 ms)
[----------] 18 tests from AuthTokenIssuingTest (24224 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:52 dist-test-slave-zkpd krb5kdc[22162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:52 dist-test-slave-zkpd krb5kdc[22162](info): set up 2 sockets
Aug 08 02:26:52 dist-test-slave-zkpd krb5kdc[22162](info): commencing operation
krb5kdc: starting...
W20260808 02:26:53.364102 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:53.480760 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38603
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:38603,127.30.160.61:42623,127.30.160.60:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:53.562023 22175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:26:53.562278 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:53.562309 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:53.562381 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:53.563495 22175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:53.563566 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:53.563591 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:53.563612 22175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:53.563642 22175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:53.565163 22175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:38603,127.30.160.61:42623,127.30.160.60:34631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:38603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:53.565449 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:53.565686 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:53.570488 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:53.570461 22183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:53.570470 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:53.570731 22175 server_base.cc:1034] running on GCE node
I20260808 02:26:53.570986 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:53.571285 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:53.572443 22175 hybrid_clock.cc:648] HybridClock initialized: now 1786156013572435 us; error 35 us; skew 500 ppm
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:53.574947 22175 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:53.575361 22175 webserver.cc:460] Webserver started at http://127.30.160.62:44579/ using document root <none> and password file <none>
I20260808 02:26:53.575596 22175 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:53.575660 22175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:53.575798 22175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:53.576879 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "6ff8614ec3174415b9c227a7dcc82fb0"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "d3d67741f89952b302fefdf5bcf7f514"
server_key_iv: "9d191efbca5c75040715eb757b652577"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.577236 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "6ff8614ec3174415b9c227a7dcc82fb0"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "d3d67741f89952b302fefdf5bcf7f514"
server_key_iv: "9d191efbca5c75040715eb757b652577"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.578536 22175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:26:53.579247 22190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:53.579423 22175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:26:53.579523 22175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "6ff8614ec3174415b9c227a7dcc82fb0"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "d3d67741f89952b302fefdf5bcf7f514"
server_key_iv: "9d191efbca5c75040715eb757b652577"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.579603 22175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:53.589269 22175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:53.590969 22175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:53.591120 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:53.594944 22175 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:38603
I20260808 02:26:53.595005 22244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:38603 every 8 connection(s)
I20260808 02:26:53.595345 22175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:53.596369 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:53.597841 22245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:53.605302 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22175
I20260808 02:26:53.605432 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:53.605603 31360 external_mini_cluster.cc:1428] Setting key f9fc5d6bd2b3789928d4d7df96dddf3e
I20260808 02:26:53.607074 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.598512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.30.160.62:38603)
0808 02:26:53.598728 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.598732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.598759 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:26:53.599254 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.599256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.599272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.599346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.599843 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.600688 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.601507 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.601881 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.601964 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.603850 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.603857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.603858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.603864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.605503 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.605919 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.605922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.605923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.606003 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.606238 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.606239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.606239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.606438 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.606521 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.606722 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.606850 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260808 02:26:53.607072 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.598487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38603 (local address 127.0.0.1:46758)
0808 02:26:53.598668 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.598703 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.599129 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.599375 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.599378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.599609 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.599762 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.599765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.600772 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.600773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.601428 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.601431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.601981 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.601982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.602040 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.602456 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.602464 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.603761 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.605579 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.605584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.605586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.605827 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.606071 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.606072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.606072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.606163 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.606514 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.606519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.606579 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.606684 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.606918 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":99,"thread_start_us":97,"threads_started":1}
I20260808 02:26:53.608038 22245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:53.608357 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.608241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55110)
0808 02:26:53.608287 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.608328 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 02:26:53.608503 22191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
I20260808 02:26:53.608937 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.608845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55114)
0808 02:26:53.608892 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.608921 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 02:26:53.609051 22245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:42623: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.30.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab.
Entry for principal kudu/127.30.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab.
I20260808 02:26:53.630663 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:42623
--webserver_interface=127.30.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab
--principal=kudu/127.30.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:38603,127.30.160.61:42623,127.30.160.60:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:26:53.658370 22245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } attempt: 1
I20260808 02:26:53.658852 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.658715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55124)
0808 02:26:53.658786 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.658823 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:26:53.659255 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.659164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55128)
0808 02:26:53.659213 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.659238 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 02:26:53.659363 22245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:42623: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
W20260808 02:26:53.713043 22253 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:26:53.713270 22253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:53.713310 22253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:53.713351 22253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:53.714461 22253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:53.714524 22253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:53.714548 22253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:53.714570 22253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:53.714599 22253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:53.716130 22253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:38603,127.30.160.61:42623,127.30.160.60:34631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab
--principal=kudu/127.30.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:53.716398 22253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:53.716620 22253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:53.721346 22261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:53.721323 22258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:53.721401 22259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:53.721486 22253 server_base.cc:1034] running on GCE node
I20260808 02:26:53.721805 22253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:53.722119 22253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:53.723286 22253 hybrid_clock.cc:648] HybridClock initialized: now 1786156013723272 us; error 26 us; skew 500 ppm
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:53.725868 22253 init.cc:377] Logged in from keytab as kudu/127.30.160.61@KRBTEST.COM (short username kudu)
I20260808 02:26:53.726238 22253 webserver.cc:460] Webserver started at http://127.30.160.61:41601/ using document root <none> and password file <none>
I20260808 02:26:53.726423 22253 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:53.726462 22253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:53.726549 22253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:53.727512 22245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } attempt: 2
I20260808 02:26:53.727540 22253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data/instance:
uuid: "533b70e915ed4fc3b5203b194a35b9e7"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "8a40ce5916ddb5469e17cb347d658367"
server_key_iv: "05feb0724f7783f6d25a491ea9001702"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.727941 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.727764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55144)
0808 02:26:53.727827 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.727870 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:26:53.727994 22253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal/instance:
uuid: "533b70e915ed4fc3b5203b194a35b9e7"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "8a40ce5916ddb5469e17cb347d658367"
server_key_iv: "05feb0724f7783f6d25a491ea9001702"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.728274 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.728196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55152)
0808 02:26:53.728233 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.728258 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260808 02:26:53.728425 22245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:42623: Network error: Client connection negotiation failed: client connection to 127.30.160.61:42623: connect: Connection refused (error 111)
I20260808 02:26:53.729441 22253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:26:53.730142 22268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:53.730320 22253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:53.730392 22253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal
uuid: "533b70e915ed4fc3b5203b194a35b9e7"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "8a40ce5916ddb5469e17cb347d658367"
server_key_iv: "05feb0724f7783f6d25a491ea9001702"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.730484 22253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:53.744469 22253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:53.746733 22253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:53.746889 22253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:53.750838 22253 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.61:42623
I20260808 02:26:53.750871 22325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.61:42623 every 8 connection(s)
I20260808 02:26:53.751219 22253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
I20260808 02:26:53.752228 22326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:53.753747 22326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:53.755973 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22253
I20260808 02:26:53.756088 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-1/wal/instance
I20260808 02:26:53.756246 31360 external_mini_cluster.cc:1428] Setting key a06ae4733cf79f6cb43de11e574fa94d
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:53.763363 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.754410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38603 (local address 127.0.0.1:46770)
0808 02:26:53.754607 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.754642 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.755080 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.755303 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.755306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.755541 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.755724 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.755728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.756660 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.756661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.757445 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.757447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.757983 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.757985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.758145 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.758606 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.758618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.760197 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.762094 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.762101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.762103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.762315 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.762533 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.762534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.762534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.762628 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.762911 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.762916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.763023 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.763101 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.763213 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260808 02:26:53.763362 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.754434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.30.160.62:38603)
0808 02:26:53.754497 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.754499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.754722 (+ 223us) server_negotiation.cc:407] Connection header received
0808 02:26:53.755158 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.755159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.755172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.755226 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.755807 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.756569 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.757529 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.757864 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.757944 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.760298 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.760300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.760301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.760310 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.762012 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.762389 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.762390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.762391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.762460 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.762695 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.762696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.762696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.762842 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.762920 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.763154 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.763243 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:26:53.764001 22326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.160.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.60@KRBTEST.COM" created.
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:26:53.771705 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.764245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55166)
0808 02:26:53.764292 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.764302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.764337 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.764598 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.764599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.764731 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.764770 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.764771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.765673 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.765674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.766187 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.766188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.766643 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.766644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.766708 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.767127 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.767132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.768578 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.770631 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.770632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.770632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.770785 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.771004 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.771005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.771005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.771083 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.771362 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.771363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.771402 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.771493 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.771592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 02:26:53.771771 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.764341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.30.160.61:42623)
0808 02:26:53.764467 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.764470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.764485 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:26:53.764499 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.764500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.764510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.764584 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.764848 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.765573 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.766273 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.766564 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.766647 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.768660 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.768667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.768668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.768674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.770538 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.770874 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.770876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.770877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.770938 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.771146 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.771147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.771147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.771283 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.771368 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.771526 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.771645 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260808 02:26:53.772524 22326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:53.772908 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.772816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50748)
0808 02:26:53.772848 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.772887 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260808 02:26:53.773077 22271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
I20260808 02:26:53.773486 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.773423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50750)
0808 02:26:53.773449 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.773471 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260808 02:26:53.773612 22326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:34631: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab.
Entry for principal kudu/127.30.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab.
I20260808 02:26:53.784339 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:34631
--webserver_interface=127.30.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab
--principal=kudu/127.30.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:38603,127.30.160.61:42623,127.30.160.60:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:26:53.819653 22245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } attempt: 3
I20260808 02:26:53.822822 22326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } attempt: 1
I20260808 02:26:53.823210 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.823097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50758)
0808 02:26:53.823146 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.823173 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:26:53.823540 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.823425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50774)
0808 02:26:53.823481 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.823513 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 02:26:53.823649 22326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:34631: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:26:53.827299 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.820153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.30.160.61:42623)
0808 02:26:53.820224 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.820227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.820238 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:53.820267 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.820268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.820280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.820352 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.820678 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.821264 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.822311 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.822546 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.822605 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.824720 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.824722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.824722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.824730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.826174 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.826538 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.826539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.826540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.826597 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.826794 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.826795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.826795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.826908 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.826980 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.827141 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.827215 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:26:53.827306 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.820058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55174)
0808 02:26:53.820123 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.820139 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.820197 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.820359 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.820360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.820520 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.820592 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.820593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.821347 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.821348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.822214 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.822217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.822622 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.822623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.822704 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.823245 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.823266 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.824634 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.826275 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.826276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.826276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.826456 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.826669 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.826669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.826670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.826732 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.826985 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.826986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.827033 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.827116 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.827215 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:26:53.827638 22245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:53.827971 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.827863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50776)
0808 02:26:53.827921 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.827947 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:26:53.828286 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.828231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50780)
0808 02:26:53.828254 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.828273 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260808 02:26:53.828384 22245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:34631: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
I20260808 02:26:53.859679 22245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } attempt: 1
I20260808 02:26:53.860198 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.860053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50790)
0808 02:26:53.860131 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.860171 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:26:53.860579 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.860498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50798)
0808 02:26:53.860538 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.860562 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 02:26:53.860688 22245 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:34631: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
W20260808 02:26:53.867602 22333 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:26:53.867792 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:53.867810 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:53.867866 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:53.869057 22333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:53.869098 22333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:53.869112 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:53.869154 22333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:53.869179 22333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:53.870661 22333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:38603,127.30.160.61:42623,127.30.160.60:34631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab
--principal=kudu/127.30.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:53.870895 22333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:53.871122 22333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:53.875979 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:53.875862 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:53.875864 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:53.876119 22333 server_base.cc:1034] running on GCE node
I20260808 02:26:53.876487 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:53.876777 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:53.877918 22333 hybrid_clock.cc:648] HybridClock initialized: now 1786156013877898 us; error 34 us; skew 500 ppm
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:53.880442 22333 init.cc:377] Logged in from keytab as kudu/127.30.160.60@KRBTEST.COM (short username kudu)
I20260808 02:26:53.880864 22333 webserver.cc:460] Webserver started at http://127.30.160.60:42077/ using document root <none> and password file <none>
I20260808 02:26:53.881098 22333 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:53.881165 22333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:53.881314 22333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:53.882387 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data/instance:
uuid: "a7f032ec2aa44e189449e0c669e4216d"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "2b35c75b8a2f2173bcb4a93556883c4e"
server_key_iv: "0ad8e74afffe8c7f77f0960ed7713672"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.882777 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal/instance:
uuid: "a7f032ec2aa44e189449e0c669e4216d"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "2b35c75b8a2f2173bcb4a93556883c4e"
server_key_iv: "0ad8e74afffe8c7f77f0960ed7713672"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.884153 22333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:26:53.884888 22349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:53.885056 22333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:53.885154 22333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal
uuid: "a7f032ec2aa44e189449e0c669e4216d"
format_stamp: "Formatted at 2026-08-08 02:26:53 on dist-test-slave-zkpd"
server_key: "2b35c75b8a2f2173bcb4a93556883c4e"
server_key_iv: "0ad8e74afffe8c7f77f0960ed7713672"
server_key_version: "encryptionkey@0"
I20260808 02:26:53.885233 22333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:53.891779 22326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } attempt: 2
I20260808 02:26:53.892230 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.892119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50808)
0808 02:26:53.892170 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.892202 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 02:26:53.892544 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.892466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50824)
0808 02:26:53.892506 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.892529 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 02:26:53.892627 22326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:34631: Network error: Client connection negotiation failed: client connection to 127.30.160.60:34631: connect: Connection refused (error 111)
I20260808 02:26:53.894070 22333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:53.895812 22333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:53.895985 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:53.899698 22333 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.60:34631
I20260808 02:26:53.899758 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.60:34631 every 8 connection(s)
I20260808 02:26:53.900141 22333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
I20260808 02:26:53.901109 22405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:53.902729 22405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:53.909045 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22333
I20260808 02:26:53.909150 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155746594248-31360-0/minicluster-data/master-2/wal/instance
I20260808 02:26:53.909324 31360 external_mini_cluster.cc:1428] Setting key 011fed71a0050b59969e831f7ca21664
I20260808 02:26:53.911207 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.903372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.30.160.62:38603)
0808 02:26:53.903448 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.903451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.903661 (+ 210us) server_negotiation.cc:407] Connection header received
0808 02:26:53.904155 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.904156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.904167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.904229 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.904765 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.905226 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.906134 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.906308 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.906358 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.908366 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.908367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.908368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.908373 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.909835 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.910213 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.910214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.910214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.910267 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.910544 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.910545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.910546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.910696 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.910775 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.911021 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.911108 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:26:53.911303 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.903336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:38603 (local address 127.0.0.1:46780)
0808 02:26:53.903526 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.903579 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.904072 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.904276 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.904279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.904511 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.904696 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.904700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.905300 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.905301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.906063 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.906065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.906384 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.906386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.906492 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.906957 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.906964 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.908290 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.909936 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.909941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.909942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.910146 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.910341 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.910342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.910342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.910477 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.910769 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.910774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.910889 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.910967 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.911076 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260808 02:26:53.911844 22405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:26:53.919085 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.912253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.30.160.61:42623)
0808 02:26:53.912312 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.912318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.912325 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:26:53.912346 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.912346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.912354 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.912397 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.912687 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.913231 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.914075 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.914270 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.914321 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.916513 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.916514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.916514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.916521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.917932 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.918257 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.918258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.918258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.918321 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.918528 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.918529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.918529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.918656 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.918731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.918889 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.918981 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:26:53.919111 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.912102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:42623 (local address 127.0.0.1:55188)
0808 02:26:53.912149 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.912158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.912194 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.912425 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.912426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.912567 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.912616 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.912618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.913306 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.913307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.913996 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.913999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.914344 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.914345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.914429 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.914746 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.914751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.916440 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.917996 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.917996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.917997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.918190 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.918390 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.918391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.918391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.918460 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.918715 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.918732 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.918775 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.918847 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.918958 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 02:26:53.919258 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.909741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.30.160.60:34631)
0808 02:26:53.910243 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.910247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.910258 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:53.910273 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.910274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.910315 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.910389 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.911145 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.912125 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.913032 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.913469 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.913520 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.916009 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.916017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.916018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.916025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.917818 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.918265 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.918268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.918268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.918322 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.918527 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.918528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.918528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.918686 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.918839 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.918923 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.919032 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":480,"thread_start_us":89,"threads_started":1}
I20260808 02:26:53.919476 22405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:53.926270 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.919777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50852)
0808 02:26:53.919835 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.919846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.919886 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.920049 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.920050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.920194 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.920231 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.920233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.921020 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.921021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.921541 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.921543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.921922 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.921922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.921976 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.922307 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.922312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.923454 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.925138 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.925139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.925139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.925362 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.925594 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.925594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.925595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.925675 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.925916 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.925917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.925964 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.926037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.926137 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 02:26:53.926312 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.919841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.30.160.60:34631)
0808 02:26:53.919929 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.919931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.919954 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:26:53.919972 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.919973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.919982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.920030 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.920321 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.920944 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.921613 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.921854 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.921901 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.923544 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.923545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.923545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.923551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.925062 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.925468 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.925469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.925469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.925530 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.925748 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.925749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.925749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.925846 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.925898 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.926071 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.926200 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":37}
I20260808 02:26:53.928225 22405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: Bootstrap starting.
I20260808 02:26:53.928637 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.920547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.30.160.62:38603)
0808 02:26:53.920583 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.920585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.920595 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:53.920618 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.920619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.920627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.920691 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.921151 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.921602 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.922451 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.922627 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.922685 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.925223 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.925225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.925225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.925232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.926865 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.927148 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.927150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.927150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.927215 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.928116 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.928117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.928117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.928224 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.928418 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.928457 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.928527 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:53.929193 22405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:53.929595 22405 log.cc:824] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:53.930352 22405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: No bootstrap required, opened a new log
I20260808 02:26:53.932255 22405 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.932374 22405 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f032ec2aa44e189449e0c669e4216d, State: Initialized, Role: FOLLOWER
I20260808 02:26:53.932454 22405 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.932757 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:53.932816 22405 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:53.932863 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
W20260808 02:26:53.934633 22424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:53.934729 22424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:53.935855 22245 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } attempt: 2
I20260808 02:26:53.937880 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.929428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.30.160.61:42623)
0808 02:26:53.929465 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.929467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.929475 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:53.929546 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.929546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.929554 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.929590 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.930133 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.930661 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.931529 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.931766 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.931838 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.934376 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.934377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.934378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.934385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.936330 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.936743 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.936744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.936744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.936812 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.937009 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.937010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.937010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.937134 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.937292 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.937712 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.937800 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:53.935142 22333 master_runner.cc:186] Error getting master registration for 127.30.160.62:38603: OK, instance_id { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" instance_seqno: 1786156013593932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:26:53.938275 22333 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:26:53.939292 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:26:53.940279 22218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ff8614ec3174415b9c227a7dcc82fb0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786
I20260808 02:26:53.944020 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.936145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50866)
0808 02:26:53.936224 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.936234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.936273 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.936452 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.936453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.936610 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.936648 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.936650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.937513 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.937514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.938710 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.938713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.939216 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.939218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.939274 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.939743 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.939749 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.941145 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.942933 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.942934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.942934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.943149 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.943348 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.943348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.943349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.943404 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.943705 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.943706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.943749 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.943816 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.943924 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 02:26:53.944063 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.936178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.30.160.60:34631)
0808 02:26:53.936336 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.936338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.936345 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:26:53.936362 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.936363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.936371 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.936414 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.936737 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.937420 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.938815 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.939104 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.939166 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.941236 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.941238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.941239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.941247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.942816 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.943222 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.943223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.943224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.943282 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.943478 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.943479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.943479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.943633 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.943705 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.943855 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.943974 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260808 02:26:53.945756 22245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0: Bootstrap starting.
I20260808 02:26:53.946508 22245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:53.946806 22245 log.cc:824] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:53.947635 22245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0: No bootstrap required, opened a new log
I20260808 02:26:53.948993 22245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.949132 22245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff8614ec3174415b9c227a7dcc82fb0, State: Initialized, Role: FOLLOWER
I20260808 02:26:53.949211 22245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.949518 22245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:53.949568 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:26:53.949502 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:53.949627 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:53.950385 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.950870 22378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a7f032ec2aa44e189449e0c669e4216d"
I20260808 02:26:53.951012 22378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [term 0 FOLLOWER]: Advancing to term 1
W20260808 02:26:53.951455 22438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:53.951512 22438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:53.952049 22378 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff8614ec3174415b9c227a7dcc82fb0 in term 1.
I20260808 02:26:53.952311 22192 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff8614ec3174415b9c227a7dcc82fb0, a7f032ec2aa44e189449e0c669e4216d; no voters:
I20260808 02:26:53.952210 22218 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [CANDIDATE]: Term 1 election: Requested vote from peers 533b70e915ed4fc3b5203b194a35b9e7 (127.30.160.61:42623), a7f032ec2aa44e189449e0c669e4216d (127.30.160.60:34631)
I20260808 02:26:53.953289 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:53.953364 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:53.953415 22427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff8614ec3174415b9c227a7dcc82fb0, State: Running, Role: LEADER
I20260808 02:26:53.953503 22175 master_runner.cc:186] Error getting master registration for 127.30.160.61:42623: OK, instance_id { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" instance_seqno: 1786156013749776 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:26:53.955719 22175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:26:53.953496 22427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.956451 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff8614ec3174415b9c227a7dcc82fb0. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:53.956549 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:53.956781 22446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:53.957139 22446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:53.959939 22378 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [term 1 FOLLOWER]: Refusing update from remote peer 6ff8614ec3174415b9c227a7dcc82fb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:53.960289 22440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:53.961575 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff8614ec3174415b9c227a7dcc82fb0. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:53.961666 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:53.962838 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:53.962915 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:53.963332 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:53.963418 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:53.963403 22447 mvcc.cc:205] Tried to move back new op lower bound from 7316095033177509888 to 7316095033163759616. Current Snapshot: MvccSnapshot[applied={T|T < 7316095033177509888}]
I20260808 02:26:53.964371 22452 mvcc.cc:205] Tried to move back new op lower bound from 7316095033177509888 to 7316095033163759616. Current Snapshot: MvccSnapshot[applied={T|T < 7316095033177509888}]
I20260808 02:26:53.965413 22446 catalog_manager.cc:1347] Generated new cluster ID: b4e410ab7e9648a4a8009269b7fa88b0
I20260808 02:26:53.965512 22446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:53.978713 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.960633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.30.160.60:34631)
0808 02:26:53.960676 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.960678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.968025 (+ 7347us) server_negotiation.cc:407] Connection header received
0808 02:26:53.968144 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.968145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.968157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.968219 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.969577 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.970254 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.972665 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.972968 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.973007 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.975331 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.975332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.975333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.975340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.977038 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.977883 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.977884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.977884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.977951 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.978149 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.978150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.978150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.978274 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.978343 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.978542 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.978618 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:26:53.981181 22446 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:53.981442 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.960139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.30.160.61:42623)
0808 02:26:53.960183 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.960185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.968217 (+ 8032us) server_negotiation.cc:407] Connection header received
0808 02:26:53.968362 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.968363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.968375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.968407 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.969154 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.969863 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.970823 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.971153 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.971196 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.973030 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.973032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.973032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.973039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.974908 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.977179 (+ 2271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.977181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.977181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.977260 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.980732 (+ 3472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.980733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.980733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.980892 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.980974 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.981263 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.981346 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:26:53.982647 22446 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:53.983697 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.960388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.30.160.62:38603)
0808 02:26:53.960441 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.960443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.960956 (+ 513us) server_negotiation.cc:407] Connection header received
0808 02:26:53.961110 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.961111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.961133 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.961163 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.962478 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.963216 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.965057 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.965353 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.965419 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.977493 (+ 12074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.977494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.977495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.977503 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.979308 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.982589 (+ 3281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.982590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.982591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.982659 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.982972 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.982973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.982974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.983096 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.983184 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.983499 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.983612 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:26:53.983711 22326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } attempt: 3
Aug 08 02:26:53 dist-test-slave-zkpd krb5kdc[22162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156013, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:26:53.992004 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.984093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:34631 (local address 127.0.0.1:50892)
0808 02:26:53.984134 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:53.984145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:53.984188 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:53.984349 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:53.984350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:53.984505 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:53.984545 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.984547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.985470 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.985471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.986271 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:53.986273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.986807 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.986808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:53.986876 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.987341 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:53.987358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:53.988870 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:53.990838 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.990839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.990839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.991052 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.991283 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:53.991284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:53.991285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:53.991357 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:53.991658 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:53.991659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:53.991711 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:26:53.991770 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:26:53.991866 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 02:26:53.992393 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:53.984023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.30.160.60:34631)
0808 02:26:53.984058 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:53.984060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:53.984241 (+ 181us) server_negotiation.cc:407] Connection header received
0808 02:26:53.984260 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:53.984261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:53.984270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:53.984313 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:53.984626 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.985377 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.986378 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:53.986682 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:53.986738 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:53.988970 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:53.988971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:53.988971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:53.988978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:53.990722 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.991139 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.991140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.991140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.991205 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:53.991432 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:53.991433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:53.991434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:53.991565 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:53.991621 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:53.992190 (+ 569us) server_negotiation.cc:299] Negotiation successful
0808 02:26:53.992297 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:53.993889 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: Bootstrap starting.
I20260808 02:26:53.994613 22326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:53.994973 22326 log.cc:824] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:53.996495 22446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0: Generated new TSK 0
I20260808 02:26:53.996708 22446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:53.997376 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: No bootstrap required, opened a new log
I20260808 02:26:53.999390 22326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.999522 22326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533b70e915ed4fc3b5203b194a35b9e7, State: Initialized, Role: FOLLOWER
I20260808 02:26:53.999596 22326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } }
I20260808 02:26:53.999965 22326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:54.000041 22297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:54.000932 22297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [term 1 FOLLOWER]: Refusing update from remote peer 6ff8614ec3174415b9c227a7dcc82fb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:54.001236 22427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:54.001434 22298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "533b70e915ed4fc3b5203b194a35b9e7"
I20260808 02:26:53.999926 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:54.008625 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:54.008805 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff8614ec3174415b9c227a7dcc82fb0. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:54.008865 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:54.008895 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:54.008944 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:26:54.009500 22469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:54.009553 22469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:54.010095 22471 mvcc.cc:205] Tried to move back new op lower bound from 7316095033314136064 to 7316095033163759616. Current Snapshot: MvccSnapshot[applied={T|T < 7316095033177509888}]
I20260808 02:26:54.010708 22298 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ff8614ec3174415b9c227a7dcc82fb0 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260808 02:26:54.032372 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.030689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.30.160.61:42623)
0808 02:26:54.030737 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.030739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.031600 (+ 861us) server_negotiation.cc:407] Connection header received
0808 02:26:54.031712 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.031713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.031725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.031759 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.032304 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55220: BlockingRecv error: recv got EOF from 127.0.0.1:55220 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 02:26:54.033294 22249 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.030953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.30.160.62:38603)
0808 02:26:54.030995 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.030997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.032696 (+ 1699us) server_negotiation.cc:407] Connection header received
0808 02:26:54.032798 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.032799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.032811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.032842 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.033230 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46800: BlockingRecv error: recv got EOF from 127.0.0.1:46800 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 02:26:54.034166 22410 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.031322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.30.160.60:34631)
0808 02:26:54.031362 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.031363 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.033527 (+ 2164us) server_negotiation.cc:407] Connection header received
0808 02:26:54.033743 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.033745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.033756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.033787 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.034094 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50898: BlockingRecv error: recv got EOF from 127.0.0.1:50898 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 02:26:54.036386 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.035641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.30.160.61:42623)
0808 02:26:54.035672 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.035674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.035859 (+ 185us) server_negotiation.cc:407] Connection header received
0808 02:26:54.035996 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.035997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.036014 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.036044 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.036342 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55228: BlockingRecv error: recv got EOF from 127.0.0.1:55228 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 02:26:54.039716 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.038978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.30.160.61:42623)
0808 02:26:54.039010 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.039012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.039212 (+ 200us) server_negotiation.cc:407] Connection header received
0808 02:26:54.039316 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.039317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.039327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.039357 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.039675 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55244: BlockingRecv error: recv got EOF from 127.0.0.1:55244 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 02:26:54.043567 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.042874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.30.160.61:42623)
0808 02:26:54.042906 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.042907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.043099 (+ 192us) server_negotiation.cc:407] Connection header received
0808 02:26:54.043190 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.043191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.043200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.043228 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.043529 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55260: BlockingRecv error: recv got EOF from 127.0.0.1:55260 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 02:26:54.049453 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.048678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.30.160.61:42623)
0808 02:26:54.048709 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.048710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.048903 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:26:54.048994 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.048995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.049003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.049030 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.049411 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55270: BlockingRecv error: recv got EOF from 127.0.0.1:55270 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 02:26:54.059401 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.058638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.30.160.61:42623)
0808 02:26:54.058673 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.058675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.058883 (+ 208us) server_negotiation.cc:407] Connection header received
0808 02:26:54.058975 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.058976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.058986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.059014 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.059352 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55278: BlockingRecv error: recv got EOF from 127.0.0.1:55278 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 02:26:54.077605 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.076711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.30.160.61:42623)
0808 02:26:54.076751 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.076753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.076994 (+ 241us) server_negotiation.cc:407] Connection header received
0808 02:26:54.077092 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.077093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.077106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.077143 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.077547 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55290: BlockingRecv error: recv got EOF from 127.0.0.1:55290 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 02:26:54.112124 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.111146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.30.160.61:42623)
0808 02:26:54.111191 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.111193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.111476 (+ 283us) server_negotiation.cc:407] Connection header received
0808 02:26:54.111582 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.111583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.111595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.111625 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.112063 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55296: BlockingRecv error: recv got EOF from 127.0.0.1:55296 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 02:26:54.178399 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.177712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.30.160.61:42623)
0808 02:26:54.177772 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.177774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.177886 (+ 112us) server_negotiation.cc:407] Connection header received
0808 02:26:54.177936 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.177937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.177947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.178011 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.178319 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55312: BlockingRecv error: recv got EOF from 127.0.0.1:55312 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 02:26:54.308558 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.307647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.30.160.61:42623)
0808 02:26:54.307748 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.307751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.307882 (+ 131us) server_negotiation.cc:407] Connection header received
0808 02:26:54.308000 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.308001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.308014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.308113 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.308490 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55314: BlockingRecv error: recv got EOF from 127.0.0.1:55314 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:26:54.472394 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff8614ec3174415b9c227a7dcc82fb0" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 38603 } } peers { permanent_uuid: "533b70e915ed4fc3b5203b194a35b9e7" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 42623 } } peers { permanent_uuid: "a7f032ec2aa44e189449e0c669e4216d" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 34631 } } }
I20260808 02:26:54.472541 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff8614ec3174415b9c227a7dcc82fb0 [sys.catalog]: This master's current role is: LEADER
W20260808 02:26:54.566535 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:54.565831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.30.160.61:42623)
0808 02:26:54.565906 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:54.565908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:54.565962 (+ 54us) server_negotiation.cc:407] Connection header received
0808 02:26:54.566017 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:54.566019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:54.566030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:54.566084 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:54.566450 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55326: BlockingRecv error: recv got EOF from 127.0.0.1:55326 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:26:54.935245 22424 catalog_manager.cc:1259] Loaded cluster ID: b4e410ab7e9648a4a8009269b7fa88b0
I20260808 02:26:54.935314 22424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: loading cluster ID for follower catalog manager: success
I20260808 02:26:54.936201 22424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: acquiring CA information for follower catalog manager: success
I20260808 02:26:54.936488 22424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7f032ec2aa44e189449e0c669e4216d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:26:55.010058 22469 catalog_manager.cc:1259] Loaded cluster ID: b4e410ab7e9648a4a8009269b7fa88b0
I20260808 02:26:55.010133 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: loading cluster ID for follower catalog manager: success
I20260808 02:26:55.010872 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: acquiring CA information for follower catalog manager: success
I20260808 02:26:55.011123 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 533b70e915ed4fc3b5203b194a35b9e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:26:55.082465 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:55.079621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.30.160.61:42623)
0808 02:26:55.079758 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:55.079761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:55.079818 (+ 57us) server_negotiation.cc:407] Connection header received
0808 02:26:55.079933 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:55.079934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:55.080001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:55.080082 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:55.080250 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.080768 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.081536 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.081809 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.081902 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:55.082097 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:55.082221 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:55.082311 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:55.082334 (+ 23us) server_negotiation.cc:299] Negotiation successful
0808 02:26:55.082357 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260808 02:26:55.086887 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:55.084585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.30.160.60:34631)
0808 02:26:55.084715 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:55.084717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:55.084727 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:55.084773 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:55.084774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:55.084808 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:55.084877 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:55.084965 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.085502 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.086203 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.086388 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.086443 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:55.086568 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:55.086654 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:55.086705 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:55.086745 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 02:26:55.086762 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260808 02:26:55.091615 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:55.088595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.30.160.61:42623)
0808 02:26:55.088630 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:55.088631 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:55.088762 (+ 131us) server_negotiation.cc:407] Connection header received
0808 02:26:55.088852 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:55.088853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:55.088863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:55.088912 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:55.089035 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.089775 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.090916 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.091116 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.091198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:55.091352 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:55.091387 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:55.091446 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:55.091509 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 02:26:55.091529 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:55.091882 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:55.088595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.30.160.62:38603)
0808 02:26:55.088704 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:55.088706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:55.088729 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:26:55.088817 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:55.088818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:55.088869 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:55.088955 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:55.089184 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.089930 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.090903 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.091219 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.091280 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:55.091492 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:55.091585 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:55.091636 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:55.091783 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 02:26:55.091805 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":54,"threads_started":1}
I20260808 02:26:55.092530 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:55.088819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.30.160.60:34631)
0808 02:26:55.088855 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:55.088857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:55.089054 (+ 197us) server_negotiation.cc:407] Connection header received
0808 02:26:55.089153 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:55.089154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:55.089163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:55.089189 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:55.089339 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.089787 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.091722 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:55.092022 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:55.092069 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:55.092266 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:55.092297 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:55.092349 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:55.092436 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 02:26:55.092462 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:26:55.093693 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22175
I20260808 02:26:55.101246 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22253
I20260808 02:26:55.107740 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22333
2026-08-08T02:26:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2797 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:55 dist-test-slave-zkpd krb5kdc[22561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:55 dist-test-slave-zkpd krb5kdc[22561](info): set up 2 sockets
Aug 08 02:26:55 dist-test-slave-zkpd krb5kdc[22561](info): commencing operation
krb5kdc: starting...
W20260808 02:26:56.172307 31360 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/test-user.keytab.
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
Entry for principal kudu/127.30.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab.
I20260808 02:26:56.294004 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34677
--webserver_interface=127.30.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:34677,127.30.160.61:33319,127.30.160.60:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:56.376727 22575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:26:56.376932 22575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:56.376950 22575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:56.376968 22575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:56.378199 22575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:56.378238 22575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:56.378252 22575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:56.378265 22575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:56.378314 22575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:56.379798 22575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:34677,127.30.160.61:33319,127.30.160.60:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.62.keytab
--principal=kudu/127.30.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.62:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:56.380089 22575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:56.380333 22575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:56.385026 22581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:56.385053 22583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:56.385056 22580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:56.385318 22575 server_base.cc:1034] running on GCE node
I20260808 02:26:56.385537 22575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:56.385774 22575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:56.386909 22575 hybrid_clock.cc:648] HybridClock initialized: now 1786156016386898 us; error 26 us; skew 500 ppm
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:56.389359 22575 init.cc:377] Logged in from keytab as kudu/127.30.160.62@KRBTEST.COM (short username kudu)
I20260808 02:26:56.389850 22575 webserver.cc:460] Webserver started at http://127.30.160.62:35761/ using document root <none> and password file <none>
I20260808 02:26:56.390087 22575 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:56.390144 22575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:56.390266 22575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:56.391316 22575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data/instance:
uuid: "b8177b7dde594c5abead810c5d9d9897"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "a48113b0058343c1c0d0321ae49f5e09"
server_key_iv: "1293f2f5630810f40cded817c374f5ab"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.391659 22575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal/instance:
uuid: "b8177b7dde594c5abead810c5d9d9897"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "a48113b0058343c1c0d0321ae49f5e09"
server_key_iv: "1293f2f5630810f40cded817c374f5ab"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.392997 22575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:56.393713 22590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:56.393919 22575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:56.394011 22575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal
uuid: "b8177b7dde594c5abead810c5d9d9897"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "a48113b0058343c1c0d0321ae49f5e09"
server_key_iv: "1293f2f5630810f40cded817c374f5ab"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.394085 22575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:56.405895 22575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:56.407608 22575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:56.407758 22575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:56.411809 22575 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.62:34677
I20260808 02:26:56.411875 22645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.62:34677 every 8 connection(s)
I20260808 02:26:56.412222 22575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:56.413156 22646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:56.414654 22646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:56.418567 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22575
I20260808 02:26:56.418681 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-0/wal/instance
I20260808 02:26:56.418877 31360 external_mini_cluster.cc:1428] Setting key 8eab399a2fa969ebeafa1830ceb57423
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:56.424382 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.415277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34677 (local address 127.0.0.1:52784)
0808 02:26:56.415477 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.415510 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.416014 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.416267 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.416269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.416507 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.416652 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.416656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.417619 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.417620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.418261 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.418262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.418774 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.418775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.418853 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.419349 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.419365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.420869 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.422944 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.422951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.422952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.423133 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.423421 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.423422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.423422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.423516 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.423989 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.423994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.424048 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.424117 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.424222 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260808 02:26:56.424500 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.415323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.30.160.62:34677)
0808 02:26:56.415545 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.415548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.415573 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:26:56.416143 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.416145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.416160 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.416235 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.416732 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.417537 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.418332 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.418637 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.418757 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.420969 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.420977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.420977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.420983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.422854 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.423215 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.423219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.423219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.423341 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.423586 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.423587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.423588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.423881 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.423987 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.424191 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.424313 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260808 02:26:56.425282 22646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:56.425614 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.425497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51906)
0808 02:26:56.425543 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.425585 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 02:26:56.425774 22593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
I20260808 02:26:56.426225 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.426117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51920)
0808 02:26:56.426155 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.426209 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 02:26:56.426353 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:33319: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.30.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab.
Entry for principal kudu/127.30.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab.
I20260808 02:26:56.446025 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:33319
--webserver_interface=127.30.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab
--principal=kudu/127.30.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:34677,127.30.160.61:33319,127.30.160.60:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:26:56.475559 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } attempt: 1
I20260808 02:26:56.476049 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.475914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51930)
0808 02:26:56.475986 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.476019 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 02:26:56.476424 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.476327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51932)
0808 02:26:56.476372 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.476405 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 02:26:56.476509 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:33319: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
W20260808 02:26:56.526995 22653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:26:56.527196 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:56.527213 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:56.527232 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:56.528389 22653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:56.528430 22653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:56.528445 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:56.528457 22653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:56.528469 22653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:56.529919 22653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:34677,127.30.160.61:33319,127.30.160.60:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.61.keytab
--principal=kudu/127.30.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.61:33319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:56.530181 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:56.530402 22653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:56.535050 22662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:56.535051 22658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:56.535087 22659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:56.535669 22653 server_base.cc:1034] running on GCE node
I20260808 02:26:56.536011 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:56.536341 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:56.537482 22653 hybrid_clock.cc:648] HybridClock initialized: now 1786156016537463 us; error 34 us; skew 500 ppm
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:56.539971 22653 init.cc:377] Logged in from keytab as kudu/127.30.160.61@KRBTEST.COM (short username kudu)
I20260808 02:26:56.540444 22653 webserver.cc:460] Webserver started at http://127.30.160.61:43371/ using document root <none> and password file <none>
I20260808 02:26:56.540690 22653 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:56.540758 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:56.540912 22653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:56.541941 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data/instance:
uuid: "f468c9d367554d778a3f37dee62014c4"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "bdc50a792919a4d07919afa2713e794b"
server_key_iv: "6788ea5fba57a91a5d80c276e7b70c4f"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.542311 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal/instance:
uuid: "f468c9d367554d778a3f37dee62014c4"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "bdc50a792919a4d07919afa2713e794b"
server_key_iv: "6788ea5fba57a91a5d80c276e7b70c4f"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.543622 22653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:56.544395 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:56.544569 22653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:56.544622 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } attempt: 2
I20260808 02:26:56.544669 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal
uuid: "f468c9d367554d778a3f37dee62014c4"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "bdc50a792919a4d07919afa2713e794b"
server_key_iv: "6788ea5fba57a91a5d80c276e7b70c4f"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.544770 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:56.545039 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.544921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51944)
0808 02:26:56.544974 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.545003 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 02:26:56.545370 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.545260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51946)
0808 02:26:56.545322 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.545348 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:26:56.545454 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.61:33319: Network error: Client connection negotiation failed: client connection to 127.30.160.61:33319: connect: Connection refused (error 111)
I20260808 02:26:56.555649 22653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:56.557507 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:56.557668 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:56.561777 22653 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.61:33319
I20260808 02:26:56.561810 22722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.61:33319 every 8 connection(s)
I20260808 02:26:56.562150 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/data/info.pb
I20260808 02:26:56.563112 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:56.564644 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:56.570803 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22653
I20260808 02:26:56.570889 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-1/wal/instance
I20260808 02:26:56.571180 31360 external_mini_cluster.cc:1428] Setting key 97ef205303338efa533385885b145361
I20260808 02:26:56.573360 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.565285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.30.160.62:34677)
0808 02:26:56.565365 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.565367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.565536 (+ 169us) server_negotiation.cc:407] Connection header received
0808 02:26:56.565968 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.565969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.565977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.566035 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.566597 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.567067 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.568053 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.568242 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.568297 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.570283 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.570284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.570285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.570290 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.571766 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.572291 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.572292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.572293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.572350 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.572622 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.572623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.572623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.572724 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.572775 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.573127 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.573216 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:26:56.573396 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.565235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34677 (local address 127.0.0.1:52794)
0808 02:26:56.565402 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.565443 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.565886 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.566092 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.566095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.566338 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.566520 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.566523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.567144 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.567145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.567975 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.567977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.568319 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.568320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.568420 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.568908 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.568925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.570208 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.571851 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.571858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.571861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.572206 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.572414 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.572415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.572416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.572556 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.572789 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.572795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.572961 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.573087 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.573240 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260808 02:26:56.574149 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.160.60@KRBTEST.COM; defaulting to no policy
I20260808 02:26:56.581143 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.574399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51956)
0808 02:26:56.574444 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.574454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.574488 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.574734 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.574734 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.574874 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.574915 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.574916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.575818 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.575819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.576364 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.576366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.576842 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.576843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.576908 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.577278 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.577282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.578409 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.580145 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.580146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.580147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.580308 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.580518 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.580519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.580519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.580572 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.580844 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.580845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.580881 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.580941 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.581035 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 02:26:56.581211 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.574468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.30.160.61:33319)
0808 02:26:56.574609 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.574613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.574629 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:26:56.574645 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.574646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.574656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.574733 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.574988 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.575704 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.576458 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.576762 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.576835 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.578490 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.578498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.578499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.578505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.580058 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.580399 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.580402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.580402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.580455 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.580644 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.580645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.580645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.580776 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.580865 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.580979 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.581088 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260808 02:26:56.581887 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.160.60@KRBTEST.COM" created.
I20260808 02:26:56.582247 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.582137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52630)
0808 02:26:56.582177 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.582219 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 02:26:56.582376 22671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
I20260808 02:26:56.583649 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.582689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52634)
0808 02:26:56.583596 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.583625 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":880}
W20260808 02:26:56.583756 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:41035: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab.
Entry for principal kudu/127.30.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab.
I20260808 02:26:56.595858 31360 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8DZQyQ/build/release/bin/kudu
/tmp/dist-test-task8DZQyQ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:41035
--webserver_interface=127.30.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab
--principal=kudu/127.30.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.160.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.160.62:34677,127.30.160.61:33319,127.30.160.60:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:26:56.632967 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } attempt: 1
I20260808 02:26:56.633536 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.633397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52650)
0808 02:26:56.633464 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.633506 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:26:56.633932 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.633848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52662)
0808 02:26:56.633891 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.633915 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 02:26:56.634039 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:41035: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
I20260808 02:26:56.636579 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } attempt: 3
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:26:56.643620 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.636901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.30.160.61:33319)
0808 02:26:56.636986 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.636988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.637000 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:26:56.637091 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.637092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.637103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.637173 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.637506 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.638026 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.638752 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.638979 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.639056 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.641214 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.641215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.641215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.641240 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.642574 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.642947 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.642948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.642949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.642999 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.643190 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.643190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.643190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.643291 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.643342 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.643501 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.643563 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:26:56.643671 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.636842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51970)
0808 02:26:56.636904 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.636937 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.637017 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.637181 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.637182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.637348 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.637414 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.637416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.638112 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.638113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.638664 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.638667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.639053 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.639054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.639137 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.639650 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.639655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.641136 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.642655 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.642656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.642656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.642873 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.643063 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.643064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.643065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.643131 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.643354 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.643355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.643406 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.643476 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.643574 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:26:56.644228 22646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:56.644534 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.644449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52670)
0808 02:26:56.644482 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.644511 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260808 02:26:56.644822 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.644763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52686)
0808 02:26:56.644789 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.644809 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260808 02:26:56.644919 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:41035: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
I20260808 02:26:56.676079 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } attempt: 1
I20260808 02:26:56.676539 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.676406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52692)
0808 02:26:56.676469 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.676509 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 02:26:56.676910 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.676804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52706)
0808 02:26:56.676855 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.676884 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 02:26:56.677007 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:41035: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
W20260808 02:26:56.680181 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:26:56.680369 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:56.680420 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:56.680461 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:56.681627 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:56.681667 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:56.681681 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:56.681715 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:56.681746 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:56.683215 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.160.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.160.62:34677,127.30.160.61:33319,127.30.160.60:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/krb5kdc/kudu_127.30.160.60.keytab
--principal=kudu/127.30.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.160.60:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:26:56.683467 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:56.683699 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:56.688333 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:56.688334 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:56.688374 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:56.688643 22731 server_base.cc:1034] running on GCE node
I20260808 02:26:56.689008 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:56.689347 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:56.690510 22731 hybrid_clock.cc:648] HybridClock initialized: now 1786156016690504 us; error 35 us; skew 500 ppm
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:56.692972 22731 init.cc:377] Logged in from keytab as kudu/127.30.160.60@KRBTEST.COM (short username kudu)
I20260808 02:26:56.693415 22731 webserver.cc:460] Webserver started at http://127.30.160.60:46155/ using document root <none> and password file <none>
I20260808 02:26:56.693665 22731 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:56.693742 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:56.693892 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:26:56.694984 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data/instance:
uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "0465ff8aa7fb768f1b9a6b61ae27cfcb"
server_key_iv: "73bb7d718d5ee6582bbb036b73dd5b13"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.695477 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal/instance:
uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "0465ff8aa7fb768f1b9a6b61ae27cfcb"
server_key_iv: "73bb7d718d5ee6582bbb036b73dd5b13"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.697052 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:26:56.697813 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:56.698009 22731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:26:56.698109 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data,/tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal
uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0"
format_stamp: "Formatted at 2026-08-08 02:26:56 on dist-test-slave-zkpd"
server_key: "0465ff8aa7fb768f1b9a6b61ae27cfcb"
server_key_iv: "73bb7d718d5ee6582bbb036b73dd5b13"
server_key_version: "encryptionkey@0"
I20260808 02:26:56.698164 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:56.702163 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } attempt: 2
I20260808 02:26:56.702559 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.702444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52718)
0808 02:26:56.702500 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.702533 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 02:26:56.702893 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.702815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52726)
0808 02:26:56.702854 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.702877 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 02:26:56.702992 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.160.60:41035: Network error: Client connection negotiation failed: client connection to 127.30.160.60:41035: connect: Connection refused (error 111)
I20260808 02:26:56.721091 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:56.722931 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:56.723037 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:56.727068 22731 rpc_server.cc:307] RPC server started. Bound to: 127.30.160.60:41035
I20260808 02:26:56.727105 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.160.60:41035 every 8 connection(s)
I20260808 02:26:56.727485 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/data/info.pb
I20260808 02:26:56.728621 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:56.730206 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:56.730751 31360 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8DZQyQ/build/release/bin/kudu as pid 22731
I20260808 02:26:56.730891 31360 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8DZQyQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155746594248-31360-0/minicluster-data/master-2/wal/instance
I20260808 02:26:56.731139 31360 external_mini_cluster.cc:1428] Setting key 2e4fd5a08dd15ca531b0414b840de5e1
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:26:56.740010 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.730945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.30.160.62:34677)
0808 02:26:56.731003 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.731005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.731271 (+ 266us) server_negotiation.cc:407] Connection header received
0808 02:26:56.731996 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.731997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.732010 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.732087 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.732658 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.733131 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.734035 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.734228 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.734303 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.736784 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.736785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.736786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.736794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.738412 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.738963 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.738965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.738965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.739029 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.739297 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.739299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.739299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.739421 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.739496 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.739791 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.739878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 02:26:56.740186 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.730915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.62:34677 (local address 127.0.0.1:52802)
0808 02:26:56.731131 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.731179 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.731849 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.732111 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.732115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.732402 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.732552 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.732557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.733212 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.733213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.733927 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.733929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.734338 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.734340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.734451 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.734953 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.734962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.736695 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.738509 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.738518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.738520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.738820 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.739106 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.739107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.739107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.739227 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.739488 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.739495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.739635 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.739755 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.739906 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260808 02:26:56.740511 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.731545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.30.160.60:41035)
0808 02:26:56.731773 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.731778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.731830 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:26:56.731959 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.731966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.731991 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.732087 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.732733 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.733516 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.734264 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.734576 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.734667 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.737231 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.737240 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.737240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.737247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.738840 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.739221 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.739224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.739227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.739283 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.739500 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.739501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.739502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.739648 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.739819 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.739869 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.739969 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260808 02:26:56.740844 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.62@KRBTEST.COM
I20260808 02:26:56.747418 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.741113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.61:33319 (local address 127.0.0.1:51980)
0808 02:26:56.741155 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.741164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.741226 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.741386 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.741387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.741532 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.741568 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.741569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.742188 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.742189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.742663 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.742665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.743066 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.743067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.743134 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.743502 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.743506 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.744620 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.746335 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.746336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.746336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.746508 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.746756 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.746757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.746757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.746821 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.747096 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.747097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.747137 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.747236 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.747328 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 02:26:56.747423 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.741197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.30.160.61:33319)
0808 02:26:56.741268 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.741270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.741279 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:56.741303 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.741304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.741314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.741383 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.741648 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.742106 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.742763 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.742976 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.743066 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.744740 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.744742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.744742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.744749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.746253 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.746583 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.746585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.746585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.746643 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.746887 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.746888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.746889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.747001 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.747075 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.747268 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.747337 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:26:56.747859 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:26:56.749593 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.741293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.30.160.62:34677)
0808 02:26:56.741328 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.741329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.741340 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:26:56.741401 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.741402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.741411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.741465 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.741930 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.742366 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.743012 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.743217 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.743299 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.746415 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.746416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.746416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.746423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.748418 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.748783 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.748785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.748785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.748839 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.749048 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.749049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.749049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.749157 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.749324 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.749360 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.749427 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.60@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:26:56.752131 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } attempt: 2
I20260808 02:26:56.754796 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.748123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52744)
0808 02:26:56.748173 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.748184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.748243 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.748355 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.748355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.748487 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.748509 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.748510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.749179 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.749179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.749630 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.749631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.750074 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.750075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.750122 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.750505 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.750510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.751765 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.753626 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.753627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.753627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.753895 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.754167 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.754168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.754168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.754235 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.754500 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.754501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.754538 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.754601 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.754697 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 02:26:56.754870 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.748168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.30.160.60:41035)
0808 02:26:56.748201 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.748203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.748228 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:26:56.748297 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.748298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.748308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.748354 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.748626 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.749093 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.749713 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.750003 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.750074 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.751841 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.751842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.751843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.751850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.753523 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.754021 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.754022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.754022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.754084 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.754305 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.754306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.754306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.754420 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.754506 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.754632 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.754754 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":18}
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.160.61@KRBTEST.COM
I20260808 02:26:56.756935 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: Bootstrap starting.
I20260808 02:26:56.757784 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: Neither blocks nor log segments found. Creating new log.
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.62@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:26:56.758141 22803 log.cc:824] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:56.758351 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.750266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.30.160.61:33319)
0808 02:26:56.750303 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.750305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.750313 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:56.750363 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.750363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.750370 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.750439 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.750900 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.751624 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.752494 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.752768 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.752803 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.755687 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.755688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.755689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.755700 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.757119 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.757526 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.757527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.757528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.757586 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.757804 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.757805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.757805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.757943 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.758121 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.758165 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.758250 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:56.759150 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: No bootstrap required, opened a new log
I20260808 02:26:56.759572 31360 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:26:56.761242 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.752359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52760)
0808 02:26:56.752407 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.752417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.752455 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.753316 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.753317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.753467 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.753507 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.753509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.754483 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.754484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.755338 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.755341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.755839 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.755841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.756179 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.756779 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.756784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.758190 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.760155 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.760156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.760157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.760371 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.760584 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.760584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.760584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.760641 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.760942 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.760942 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.760987 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.761068 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.761157 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 02:26:56.761538 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.752537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.30.160.60:41035)
0808 02:26:56.752922 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.752924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.752932 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:26:56.752952 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.752953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.752965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.753292 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.753609 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.754400 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.755421 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.755724 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.757290 (+ 1566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.758277 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.758278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.758278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.758297 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.760061 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.760457 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.760458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.760458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.760511 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.760709 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.760709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.760709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.760865 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.760933 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.761111 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.761191 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":95,"threads_started":1}
I20260808 02:26:56.760663 22619 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b8177b7dde594c5abead810c5d9d9897"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52816
I20260808 02:26:56.761514 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.761667 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4f75e41e5b4f4f8ffeba8cee3ceea0, State: Initialized, Role: FOLLOWER
I20260808 02:26:56.761750 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.762069 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:56.762043 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.762156 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:56.762868 22646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897: Bootstrap starting.
I20260808 02:26:56.763561 22646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897: Neither blocks nor log segments found. Creating new log.
W20260808 02:26:56.763805 22821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:56.763849 22821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:56.763860 22646 log.cc:824] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:56.764700 22646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897: No bootstrap required, opened a new log
I20260808 02:26:56.766074 22646 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.766208 22646 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8177b7dde594c5abead810c5d9d9897, State: Initialized, Role: FOLLOWER
I20260808 02:26:56.766284 22646 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.766561 22646 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:56.764416 22731 master_runner.cc:186] Error getting master registration for 127.30.160.62:34677: OK, instance_id { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" instance_seqno: 1786156016410610 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:26:56.766623 22619 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:26:56.766695 22619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:56.766695 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:26:56.767455 22619 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.767666 22619 leader_election.cc:288] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [CANDIDATE]: Term 1 election: Requested vote from peers f468c9d367554d778a3f37dee62014c4 (127.30.160.61:33319), 0c4f75e41e5b4f4f8ffeba8cee3ceea0 (127.30.160.60:41035)
I20260808 02:26:56.768124 22775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8177b7dde594c5abead810c5d9d9897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0"
I20260808 02:26:56.768249 22775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:56.769132 22775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8177b7dde594c5abead810c5d9d9897 in term 1.
I20260808 02:26:56.769359 22593 leader_election.cc:302] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4f75e41e5b4f4f8ffeba8cee3ceea0, b8177b7dde594c5abead810c5d9d9897; no voters:
W20260808 02:26:56.769349 22839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:56.769748 22839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:56.766573 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.769999 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:56.770131 22841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:56.770351 22841 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [term 1 LEADER]: Becoming Leader. State: Replica: b8177b7dde594c5abead810c5d9d9897, State: Running, Role: LEADER
I20260808 02:26:56.770563 22841 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.771015 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8177b7dde594c5abead810c5d9d9897. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.771109 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:56.771988 22575 master_runner.cc:186] Error getting master registration for 127.30.160.61:33319: OK, instance_id { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" instance_seqno: 1786156016560666 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:26:56.774266 22575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:26:56.776064 22846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:56.776263 22846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:56.787537 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.768914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.30.160.62:34677)
0808 02:26:56.768961 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.768963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.770540 (+ 1577us) server_negotiation.cc:407] Connection header received
0808 02:26:56.770639 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.770640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.770650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.770679 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.778099 (+ 7420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.778843 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.780023 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.780315 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.780355 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.782613 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.782615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.782615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.782622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.784771 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.785673 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.785674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.785674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.785732 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.786078 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.786079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.786080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.786541 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.786637 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.787351 (+ 714us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.787430 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 02:26:56.789654 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.770314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.30.160.60:41035)
0808 02:26:56.770351 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.770352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.778276 (+ 7924us) server_negotiation.cc:407] Connection header received
0808 02:26:56.778528 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.778528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.778540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.778567 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.779965 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.780681 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.783478 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.783832 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.783877 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.785909 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.785911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.785911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.785919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.787659 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.788699 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.788701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.788701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.788766 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.788983 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.788984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.788984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.789123 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.789231 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.789444 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.789531 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:56.791901 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.769991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.30.160.61:33319)
0808 02:26:56.770038 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.770040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.778190 (+ 8150us) server_negotiation.cc:407] Connection header received
0808 02:26:56.778414 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.778415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.778426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.778454 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.781699 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.782463 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.784227 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.784533 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.784580 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.788654 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.788655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.788656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.788664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.790602 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.790986 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.790986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.790987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.791043 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.791264 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.791265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.791265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.791397 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.791478 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.791726 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.791805 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:26:56.791931 22775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [term 1 FOLLOWER]: Refusing update from remote peer b8177b7dde594c5abead810c5d9d9897: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:56.792390 22841 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:56.794109 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } attempt: 3
I20260808 02:26:56.794566 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8177b7dde594c5abead810c5d9d9897. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.794660 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 02:26:56 dist-test-slave-zkpd krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156016, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.160.61@KRBTEST.COM for kudu/127.30.160.60@KRBTEST.COM
I20260808 02:26:56.802022 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.794327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.160.60:41035 (local address 127.0.0.1:52788)
0808 02:26:56.794364 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:56.794375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:56.794419 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:56.794645 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:56.794646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:56.794804 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:56.794843 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.794845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.795743 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.795744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.796318 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:56.796320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.796805 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.796806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:56.796862 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.797265 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:56.797270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:56.798750 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:56.800597 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.800598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.800599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.800802 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.801009 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:56.801010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:56.801010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:56.801063 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:56.801704 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:56.801704 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:56.801771 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 02:26:56.801838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 02:26:56.801932 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260808 02:26:56.802043 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:56.794492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.30.160.60:41035)
0808 02:26:56.794527 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.794529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.794535 (+ 6us) server_negotiation.cc:407] Connection header received
0808 02:26:56.794553 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.794554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.794563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.794608 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.794929 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.795648 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.796421 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:56.796742 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:56.796804 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:56.798834 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:56.798836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:56.798836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:56.798842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:56.800531 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.800884 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.800885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.800885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.800948 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:56.801123 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:56.801135 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:56.801135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:56.801266 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:56.801327 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:56.801894 (+ 567us) server_negotiation.cc:299] Negotiation successful
0808 02:26:56.801961 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:26:56.803450 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: Bootstrap starting.
I20260808 02:26:56.804147 22723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:56.804461 22723 log.cc:824] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:56.805124 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.805208 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:56.805349 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.805421 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:56.805599 22848 mvcc.cc:205] Tried to move back new op lower bound from 7316095044775796736 to 7316095044693000192. Current Snapshot: MvccSnapshot[applied={T|T < 7316095044775796736}]
I20260808 02:26:56.805719 22847 mvcc.cc:205] Tried to move back new op lower bound from 7316095044775796736 to 7316095044693000192. Current Snapshot: MvccSnapshot[applied={T|T < 7316095044775796736}]
I20260808 02:26:56.806218 22846 catalog_manager.cc:1347] Generated new cluster ID: 78f5e1a72daa4c1c953b5fb5d717047d
I20260808 02:26:56.806270 22846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:56.812345 22846 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:56.813071 22846 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:56.823654 22846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897: Generated new TSK 0
I20260808 02:26:56.824005 22846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:26:56.824110 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: No bootstrap required, opened a new log
I20260808 02:26:56.826469 22723 raft_consensus.cc:348] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.826603 22723 raft_consensus.cc:729] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f468c9d367554d778a3f37dee62014c4, State: Initialized, Role: FOLLOWER
I20260808 02:26:56.826678 22723 consensus_queue.cc:260] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } }
I20260808 02:26:56.827023 22723 sys_catalog.cc:564] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:56.827064 22696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8177b7dde594c5abead810c5d9d9897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f468c9d367554d778a3f37dee62014c4"
I20260808 02:26:56.827147 22696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:56.828176 22696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8177b7dde594c5abead810c5d9d9897 in term 1.
I20260808 02:26:56.828415 22695 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [term 1 FOLLOWER]: Refusing update from remote peer b8177b7dde594c5abead810c5d9d9897: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:56.828814 22853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:26:56.829773 22866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:26:56.829891 22866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:26:56.826987 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.830435 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:56.830500 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8177b7dde594c5abead810c5d9d9897. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.830561 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:56.830600 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:56.830643 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:26:56.832930 22869 mvcc.cc:205] Tried to move back new op lower bound from 7316095044905918464 to 7316095044693000192. Current Snapshot: MvccSnapshot[applied={T|T < 7316095044775796736}]
W20260808 02:26:56.851198 22649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.849636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.30.160.62:34677)
0808 02:26:56.849679 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.849681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.850570 (+ 889us) server_negotiation.cc:407] Connection header received
0808 02:26:56.850670 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.850671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.850682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.850712 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.851134 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52848: BlockingRecv error: recv got EOF from 127.0.0.1:52848 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 02:26:56.851918 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.850019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.30.160.61:33319)
0808 02:26:56.850054 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.850056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.851429 (+ 1373us) server_negotiation.cc:407] Connection header received
0808 02:26:56.851523 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.851524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.851534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.851562 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.851845 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52006: BlockingRecv error: recv got EOF from 127.0.0.1:52006 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 02:26:56.852671 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.850312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.30.160.60:41035)
0808 02:26:56.850349 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.850351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.852098 (+ 1747us) server_negotiation.cc:407] Connection header received
0808 02:26:56.852288 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.852289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.852299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.852328 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.852611 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52792: BlockingRecv error: recv got EOF from 127.0.0.1:52792 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:26:56.856877 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.856155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.30.160.61:33319)
0808 02:26:56.856187 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.856189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.856409 (+ 220us) server_negotiation.cc:407] Connection header received
0808 02:26:56.856505 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.856506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.856522 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.856550 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.856835 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52022: BlockingRecv error: recv got EOF from 127.0.0.1:52022 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 02:26:56.859789 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.859072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.30.160.61:33319)
0808 02:26:56.859103 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.859104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.859334 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:26:56.859426 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.859427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.859436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.859464 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.859750 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52038: BlockingRecv error: recv got EOF from 127.0.0.1:52038 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 02:26:56.864073 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.863135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.30.160.61:33319)
0808 02:26:56.863258 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.863260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.863526 (+ 266us) server_negotiation.cc:407] Connection header received
0808 02:26:56.863623 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.863624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.863635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.863664 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.864017 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52052: BlockingRecv error: recv got EOF from 127.0.0.1:52052 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 02:26:56.869881 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.869160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.30.160.61:33319)
0808 02:26:56.869190 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.869192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.869417 (+ 225us) server_negotiation.cc:407] Connection header received
0808 02:26:56.869510 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.869511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.869520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.869547 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.869843 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52062: BlockingRecv error: recv got EOF from 127.0.0.1:52062 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 02:26:56.880030 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.879064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.30.160.61:33319)
0808 02:26:56.879111 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.879113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.879374 (+ 261us) server_negotiation.cc:407] Connection header received
0808 02:26:56.879475 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.879475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.879486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.879516 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.879860 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52064: BlockingRecv error: recv got EOF from 127.0.0.1:52064 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:26:56.898417 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.897491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.30.160.61:33319)
0808 02:26:56.897547 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.897550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.897819 (+ 269us) server_negotiation.cc:407] Connection header received
0808 02:26:56.897926 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.897927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.897940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.897969 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.898359 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52072: BlockingRecv error: recv got EOF from 127.0.0.1:52072 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 02:26:56.932761 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.931858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.30.160.61:33319)
0808 02:26:56.931902 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.931904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.932187 (+ 283us) server_negotiation.cc:407] Connection header received
0808 02:26:56.932300 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.932301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.932313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.932344 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.932705 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52084: BlockingRecv error: recv got EOF from 127.0.0.1:52084 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:26:56.999092 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:56.998237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.30.160.61:33319)
0808 02:26:56.998314 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:56.998316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:56.998454 (+ 138us) server_negotiation.cc:407] Connection header received
0808 02:26:56.998531 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:56.998532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:56.998542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:56.998594 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:56.999022 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52100: BlockingRecv error: recv got EOF from 127.0.0.1:52100 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 02:26:57.129277 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:57.128381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.30.160.61:33319)
0808 02:26:57.128462 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.128464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.128604 (+ 140us) server_negotiation.cc:407] Connection header received
0808 02:26:57.128732 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.128733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.128745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.128817 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:57.129206 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52110: BlockingRecv error: recv got EOF from 127.0.0.1:52110 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:26:57.217263 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8177b7dde594c5abead810c5d9d9897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8177b7dde594c5abead810c5d9d9897" member_type: VOTER last_known_addr { host: "127.30.160.62" port: 34677 } } peers { permanent_uuid: "f468c9d367554d778a3f37dee62014c4" member_type: VOTER last_known_addr { host: "127.30.160.61" port: 33319 } } peers { permanent_uuid: "0c4f75e41e5b4f4f8ffeba8cee3ceea0" member_type: VOTER last_known_addr { host: "127.30.160.60" port: 41035 } } }
I20260808 02:26:57.217399 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8177b7dde594c5abead810c5d9d9897 [sys.catalog]: This master's current role is: LEADER
W20260808 02:26:57.387090 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:26:57.386530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.30.160.61:33319)
0808 02:26:57.386596 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.386598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.386607 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:57.386666 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.386667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.386677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.386726 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:57.387040 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52116: BlockingRecv error: recv got EOF from 127.0.0.1:52116 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:26:57.764279 22821 catalog_manager.cc:1259] Loaded cluster ID: 78f5e1a72daa4c1c953b5fb5d717047d
I20260808 02:26:57.764365 22821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: loading cluster ID for follower catalog manager: success
I20260808 02:26:57.765162 22821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: acquiring CA information for follower catalog manager: success
I20260808 02:26:57.765471 22821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c4f75e41e5b4f4f8ffeba8cee3ceea0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:26:57.830355 22866 catalog_manager.cc:1259] Loaded cluster ID: 78f5e1a72daa4c1c953b5fb5d717047d
I20260808 02:26:57.830464 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: loading cluster ID for follower catalog manager: success
I20260808 02:26:57.831269 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: acquiring CA information for follower catalog manager: success
I20260808 02:26:57.831580 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f468c9d367554d778a3f37dee62014c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:26:57.903169 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:57.900476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.30.160.61:33319)
0808 02:26:57.900610 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.900613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.900625 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:26:57.900696 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.900697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.900767 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.900853 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:57.901012 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.901566 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.902341 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.902544 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.902614 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:57.902804 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:57.902919 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:57.903007 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:57.903046 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:26:57.903066 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260808 02:26:57.907284 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:57.904932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.30.160.60:41035)
0808 02:26:57.905069 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.905070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.905080 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:26:57.905142 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.905142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.905177 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.905248 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:57.905344 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.905877 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.906547 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.906756 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.906833 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:57.906952 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:57.907044 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:57.907101 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:57.907136 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 02:26:57.907155 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260808 02:26:57.912016 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:57.908995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.30.160.61:33319)
0808 02:26:57.909031 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.909033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.909143 (+ 110us) server_negotiation.cc:407] Connection header received
0808 02:26:57.909241 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.909242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.909253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.909307 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:57.909456 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.910086 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.911208 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.911489 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.911549 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:57.911692 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:57.911720 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:57.911769 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:57.911875 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:26:57.911929 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:26:57.912295 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:57.909351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.30.160.60:41035)
0808 02:26:57.909515 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.909517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.909526 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:26:57.909548 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.909549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.909558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.909611 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:57.909787 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.910479 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.911386 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.911698 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.911799 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:57.911994 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:57.912032 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:57.912094 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:57.912169 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:26:57.912190 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260808 02:26:57.912535 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:57.908998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.30.160.62:34677)
0808 02:26:57.909107 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:57.909110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:57.909172 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:26:57.909283 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:57.909284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:57.909336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:57.909430 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:57.909613 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.910368 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.911443 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:57.911764 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:57.911823 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:57.912084 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:57.912189 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:57.912244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:57.912433 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:26:57.912461 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260808 02:26:57.919191 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22575
I20260808 02:26:57.927157 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22653
I20260808 02:26:57.933498 31360 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8DZQyQ/build/release/bin/kudu with pid 22731
2026-08-08T02:26:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2825 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5623 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271347 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST